Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-17463

Support the switch StringTable Split ID feature

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.0, 3.3.5, 3.3.3, 3.3.4
    • None
    • namenode

    Description

      desc:


      • Hadoop 3.2 introduced optimization features for HDFS StringTable (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of Hadoop upgraded to 3.2 and later versions not supporting downgrade operations.

      • This issue has also been discussed in HDFS-14831, and it is recommended to revert the feature, but it cannot fundamentally solve the problem。

      • Therefore, we have added an optimization to support downgrading

       

      Solution:

      • First, we will add the "dfs. image. save. splitId. stringTable" conf switch "StringTable optimization feature" is enabled
      • When the conf value is false, an Image file compatible with lower versions of HDFS is generated to support downgrading.

      • The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2 is shown in the following figure.
      • With the sub-sections feature introduced in HDFS-14617, Protobuf can support compatible reading.

      • The data structure causing incompatible differences is mainly StringTable.

      • In "dfs.image.save.splitId.stringTable = false " the Id generation order of StringTable starts from 0 to Integer.Max. When true, the Id value range follows the latest rules.
         

      Attachments

        1. error.png
          171 kB
          wangzhihui
        2. Image_struct.png
          85 kB
          wangzhihui

        Issue Links

          Activity

            People

              Unassigned Unassigned
              hiwangzhihui wangzhihui
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: