Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
None
-
None
-
ghx-label-10
Description
As IMPALA-7539 is unresolved, HDFS file/dir permissions are not used at all in local catalog mode. However, in catalogd, we still load them (in single thread for a table!) which could dominant the table loading time when there are lots of partitions. Here is an example timeline for a REFRESH statement on an unloaded table:
Catalog Server Operation: 2s300ms - Got catalog version read lock: 26.407us (26.407us) - Start loading table: 314.663us (288.256us) - Got Metastore client: 629.599us (314.936us) - Fetched table from Metastore: 7.248ms (6.618ms) - Loaded table schema: 27.947ms (20.699ms) - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms) - Got access level: 1s514ms (588.314us) - Created partition builders: 2s103ms (588.270ms) - Start loading file metadata: 2s103ms (49.760us) - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms) - Async loaded table: 2s289ms (6.931ms) - Loaded table from scratch: 2s289ms (72.038us) - Got table read lock: 2s289ms (2.289us) - Finished resetMetadata request: 2s300ms (10.188ms)
The majority of the time is spent in "Preloaded permissions cache for 1824 partitions".
Currently, catalogd can skip loading HDFS permissions in local catalog mode until IMPALA-7539 is unresolved.
Attachments
Issue Links
- relates to
-
IMPALA-7539 Support HDFS permissions checks with LocalCatalog
-
- Open
-