Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7285 Erasure Coding Support inside HDFS
  3. HDFS-8254

Avoid assigning a leading streamer in StripedDataStreamer to tolerate datanode failure

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • HDFS-7285
    • None
    • None
    • Reviewed

    Description

      StripedDataStreamer javadoc is shown below.

       * The StripedDataStreamer class is used by {@link DFSStripedOutputStream}.
 * There are two kinds of StripedDataStreamer, leading streamer and ordinary
 * stream. Leading streamer requests a block group from NameNode, unwraps
 * it to located blocks and transfers each located block to its corresponding
 * ordinary streamer via a blocking queue.

      Leading streamer is the streamer with index 0. When the datanode of the leading streamer fails, the other steamers cannot continue since no one will request a block group from NameNode anymore.

      Attachments

        1. h8254_20150618.patch
          33 kB
          Tsz-wo Sze
        2. h8254_20150616.patch
          33 kB
          Tsz-wo Sze
        3. h8254_20150526b.patch
          30 kB
          Tsz-wo Sze
        4. h8254_20150526.patch
          29 kB
          Tsz-wo Sze

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HDFS-7672 Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks

          • Major - Major loss of function.
          • Resolved
          supercedes

          Sub-task - The sub-task of the issue HDFS-8366 Erasure Coding: Make the timeout parameter of polling blocking queue configurable in DFSStripedOutputStream

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              szetszwo Tsz-wo Sze
              szetszwo Tsz-wo Sze
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: