Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14004

[Replication] Inconsistency between Memstore and WAL may result in data in remote cluster that is not in the origin

    XMLWordPrintableJSON

Details

    • Reviewed
    • Hide
      Now when replicating a wal file which is still opened for write, we will get its committed length from the WAL instance in the same RS to prevent replicating uncommit WALEdit.

      This is very important if you use AsyncFSWAL, as we use fan-out in AsyncFSWAL. The data written to DN will be visible immediately as all DNs think it is the end of a pipeline, although the client has not received an ack, and also NN may truncate the file if the client crashes at the same time.
      Show
      Now when replicating a wal file which is still opened for write, we will get its committed length from the WAL instance in the same RS to prevent replicating uncommit WALEdit. This is very important if you use AsyncFSWAL, as we use fan-out in AsyncFSWAL. The data written to DN will be visible immediately as all DNs think it is the end of a pipeline, although the client has not received an ack, and also NN may truncate the file if the client crashes at the same time.

    Description

      Looks like the current write path can cause inconsistency between memstore/hfile and WAL which cause the slave cluster has more data than the master cluster.

      The simplified write path looks like:
      1. insert record into Memstore
      2. write record to WAL
      3. sync WAL
      4. rollback Memstore if 3 fails

      It's possible that the HDFS sync RPC call fails, but the data is already (may partially) transported to the DNs which finally get persisted. As a result, the handler will rollback the Memstore and the later flushed HFile will also skip this record.

      ==================================

      This is a long lived issue. The above problem is solved by write path reorder, as now we will sync wal first before modifying memstore. But the problem may still exists as replication thread may read the new data before we return from hflush. See this document for more details:

      https://docs.google.com/document/d/11AyWtGhItQs6vsLRIx32PwTxmBY3libXwGXI25obVEY/edit#

      So we need to keep a sync length in WAL and tell replication wal reader this is limit when you read this wal file.

      Attachments

        1. HBASE-14004-v3.patch
          79 kB
          Duo Zhang
        2. HBASE-14004-v2.patch
          86 kB
          Duo Zhang
        3. HBASE-14004-v2.patch
          86 kB
          Duo Zhang
        4. HBASE-14004-v1.patch
          85 kB
          Duo Zhang
        5. HBASE-14004.patch
          84 kB
          Duo Zhang

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HBASE-21503 Replication normal source can get stuck due potential race conditions between source wal reader and wal provider initialization threads.

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HBASE-18845 TestReplicationSmallTests fails after HBASE-14004

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length.

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HBASE-28184 Tailing the WAL is very slow if there are multiple peers.

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-5954 Allow proper fsync support for HBase

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-14790 Implement a new DFSOutputStream for logging WAL only

          • Major - Major loss of function.
          • Closed
          links to

          Web Link fsync behind and fsync on close (and HDFS-744)

          Web Link Review Board

          Web Link The new design for HBase writing logic to prevent data loss and inconsistency(HBASE-14004)

          Activity

            People

              zhangduo Duo Zhang
              heliangliang He Liangliang
              Votes:
              0 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: