Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15039

Cache meta file length of FinalizedReplica to reduce call File.length()

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 3.3.0
    • datanode
    • None

    Description

      When use ReplicaCachingGetSpaceUsed to get the volume space used.  It will call File.length() for every meta file of replica. That add more disk IO, we found the slow log as below. For finalized replica, the size of meta file is not changed, i think we can cache the value.

      org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas size: 1166 dfsUsed: 72227113183 on volume: DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms

      Attachments

        1. HDFS-15039.006.patch
          3 kB
          Wei-Chiu Chuang
        2. HDFS-15039.patch
          3 kB
          Yang Yun
        3. HDFS-15039.patch
          3 kB
          Yang Yun
        4. HDFS-15039.patch
          3 kB
          Yang Yun
        5. HDFS-15039.patch
          3 kB
          Yang Yun
        6. HDFS-15039.patch
          3 kB
          Yang Yun

        Issue Links

          Activity

            People

              hadoop_yangyun Yang Yun
              hadoop_yangyun Yang Yun
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: