Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14317

Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 1.2.0, 1.1.1
    • 1.2.0, 1.3.0, 2.0.0
    • wal
    • None
    • Reviewed
    • Hide
      Tighten up WAL-use semantic.

      1. If an append or a sync throws an exception, all subsequent attempts at using the log will also throw this same exception. The WAL is now a lame-duck until you roll it.
      2. If a successful append, and then we fail to sync the append, this is a fatal exception. The container must abort to replay the WAL logs even though we have told the client that the appends failed.

      The above rules have been applied laxly up to this; it used to be possible to get a good sync to go in over the top of a failed append. This has been fixed in this patch.

      Also fixed a hang in the WAL subsystem if a request to pause the write pipeline took on a failed sync. before the roll requests sync got scheduled.


      TODO: Revisit our WAL system. HBASE-12751 helps rationalize our write pipeline. In particular, it manages sequenceid inside mvcc which should make it so we can purge mechanism that writes empty, unflushed appends just to get the next sequenceid... problematic when WAL goes lame-duck. Lets get it in.
      TODO: A successful append followed by a failed sync probably only needs us replace the WAL (if we have signalled the client that the appends failed). Bummer is that replicating, these last appends might make it to the sink cluster or get replayed during recovery. HBase should keep its own WAL length? Or sequenceid of last successful sync should be passed when doing recovery and replication?
      Show
      Tighten up WAL-use semantic. 1. If an append or a sync throws an exception, all subsequent attempts at using the log will also throw this same exception. The WAL is now a lame-duck until you roll it. 2. If a successful append, and then we fail to sync the append, this is a fatal exception. The container must abort to replay the WAL logs even though we have told the client that the appends failed. The above rules have been applied laxly up to this; it used to be possible to get a good sync to go in over the top of a failed append. This has been fixed in this patch. Also fixed a hang in the WAL subsystem if a request to pause the write pipeline took on a failed sync. before the roll requests sync got scheduled. TODO: Revisit our WAL system. HBASE-12751 helps rationalize our write pipeline. In particular, it manages sequenceid inside mvcc which should make it so we can purge mechanism that writes empty, unflushed appends just to get the next sequenceid... problematic when WAL goes lame-duck. Lets get it in. TODO: A successful append followed by a failed sync probably only needs us replace the WAL (if we have signalled the client that the appends failed). Bummer is that replicating, these last appends might make it to the sink cluster or get replayed during recovery. HBase should keep its own WAL length? Or sequenceid of last successful sync should be passed when doing recovery and replication?

    Description

      hbase-1.1.1 and hadoop-2.7.1

      We try to roll logs because can't append (See HDFS-8960) but we get stuck. See attached thread dump and associated log. What is interesting is that syncers are waiting to take syncs to run and at same time we want to flush so we are waiting on a safe point but there seems to be nothing in our ring buffer; did we go to roll log and not add safe point sync to clear out ringbuffer?

      Needs a bit of study. Try to reproduce.

      Attachments

        1. [Java] RS stuck on WAL sync to a dead DN - Pastebin.com.html
          1.19 MB
          Michael Stack
        2. 14317.branch-1.txt
          83 kB
          Michael Stack
        3. 14317.branch-1.txt
          83 kB
          Michael Stack
        4. 14317.branch-1.v2.txt
          83 kB
          Michael Stack
        5. 14317.branch-1.v2.txt
          83 kB
          Michael Stack
        6. 14317.branch-1.v2.txt
          83 kB
          Michael Stack
        7. 14317.branch-1.v2.txt
          83 kB
          Michael Stack
        8. 14317.branch-1.v2.txt
          83 kB
          Michael Stack
        9. 14317.branch-1.v2.txt
          83 kB
          Michael Stack
        10. 14317.branch-1.v2.txt
          83 kB
          Michael Stack
        11. 14317.branch-1.v2.txt
          83 kB
          Michael Stack
        12. 14317.test.txt
          20 kB
          Michael Stack
        13. 14317v10.txt
          57 kB
          Michael Stack
        14. 14317v11.txt
          73 kB
          Michael Stack
        15. 14317v12.txt
          73 kB
          Michael Stack
        16. 14317v13.txt
          73 kB
          Michael Stack
        17. 14317v14.txt
          79 kB
          Michael Stack
        18. 14317v15.txt
          82 kB
          Michael Stack
        19. 14317v5.branch-1.2.txt
          43 kB
          Michael Stack
        20. 14317v5.txt
          43 kB
          Michael Stack
        21. 14317v9.txt
          50 kB
          Michael Stack
        22. append-only-test.patch
          12 kB
          Elliott Neil Clark
        23. HBASE-14317.patch
          5 kB
          Elliott Neil Clark
        24. HBASE-14317-v1.patch
          16 kB
          Elliott Neil Clark
        25. HBASE-14317-v2.patch
          17 kB
          Elliott Neil Clark
        26. HBASE-14317-v3.patch
          17 kB
          Elliott Neil Clark
        27. HBASE-14317-v4.patch
          17 kB
          Elliott Neil Clark
        28. raw.php
          174 kB
          Michael Stack
        29. repro.txt
          32 kB
          Michael Stack
        30. san_dump.txt
          225 kB
          Elliott Neil Clark
        31. subset.of.rs.log
          877 kB
          Michael Stack
        32. timeouts.branch-1.txt
          7 kB
          Michael Stack

        Issue Links

          Activity

            People

              stack Michael Stack
              stack Michael Stack
              Votes:
              0 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: