Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6871

Improve NameNode performance when creating file

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Critical
    • Resolution: Duplicate
    • None
    • 2.6.0
    • namenode, performance
    • None

    Description

      Creating file with overwrite flag will cause NN fall into flush edit logs and block other requests if the file exists.

      When we create a file with overwrite flag (default is true) in HDFS, NN will remove original file if it exists. In FSNamesystem#startFileInternal, NN already holds the write lock, it calls deleteInt if the file exists, there is logSync in deleteInt. So in this case, logSync is under write lock, it will heavily affect the NN performance.

      We should ignore the force logSync in deleteInt in this case.

      Attachments

        1. HDFS-6871.001.patch
          3 kB
          Yi Liu
        2. HDFS-6871.002.patch
          3 kB
          Yi Liu
        3. HDFS-6871.003.patch
          9 kB
          Yi Liu

        Issue Links

          Activity

            People

              hitliuyi Yi Liu
              hitliuyi Yi Liu
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: