Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
Description
When we list a large directory, we need to wait a lot of time. This is because the NameNode only returns the number of files corresponding to dfs.ls.limit each time, and then the client iteratively obtains the remaining files. But in many scenarios, we only need to know part of the files in the current directory, and then process this part of the file. After processing, go to get the remaining files. So can we add a limit on the number of files and return it to the client after obtaining the specified number of files or NameNode returnes files based on lock hold time instead of just relying on a configuration.