Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-16081

List a large directory, the client waits for a long time

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • hdfs-client
    • None

    Description

      When we list a large directory, we need to wait a lot of time. This is because the NameNode only returns the number of files corresponding to dfs.ls.limit each time, and then the client iteratively obtains the remaining files. But in many scenarios, we only need to know part of the files in the current directory, and then process this part of the file. After processing, go to get the remaining files. So can we add a limit on the number of files and return it to the client after obtaining the specified number of files  or NameNode returnes files based on lock hold time instead of just relying on a configuration. 

      Attachments

        Activity

          People

            Unassigned Unassigned
            lei w lei w
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: