Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-alpha1
    • None
    • None
    • Reviewed

    Description

      Currently existing Java RS coders based on GaloisField implementation have some drawbacks or limitations:

      • The decoder computes not really erased units unnecessarily (HADOOP-11871);
      • The decoder requires parity units + data units order for the inputs in the decode API (HADOOP-12040);
      • Need to support or align with native erasure coders regarding concrete coding algorithms and matrix, so Java coders and native coders can be easily swapped in/out and transparent to HDFS (HADOOP-12010);
      • It's unnecessarily flexible but incurs some overhead, as HDFS erasure coding is totally a byte based data system, we don't need to consider other symbol size instead of 256.

      This desires to implement another RS coder in pure Java, in addition to the existing GaliosField from HDFS-RAID. The new Java RS coder will be favored and used by default to resolve the related issues. The old HDFS-RAID originated coder will still be there for comparing, and converting old data from HDFS-RAID systems.

      Attachments

        1. HADOOP-12041-v1.patch
          43 kB
          Kai Zheng
        2. HADOOP-12041-v2.patch
          43 kB
          Kai Zheng
        3. HADOOP-12041-v3.patch
          47 kB
          Kai Zheng
        4. HADOOP-12041-v4.patch
          50 kB
          Kai Zheng
        5. HADOOP-12041-v5.patch
          51 kB
          Kai Zheng
        6. HADOOP-12041-v6.patch
          51 kB
          Kai Zheng
        7. HADOOP-12041-v7.patch
          54 kB
          Kai Zheng
        8. HADOOP-12041-v8.patch
          54 kB
          Kai Zheng

        Issue Links

          Activity

            People

              drankye Kai Zheng
              drankye Kai Zheng
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: