[HIVE-14423] S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Minor
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 2.2.0
Component/s: None
Labels:
- TODOC2.2

Target Version/s:

2.1.0

Description

When partition stats are not available in metastore, it tries to get the file sizes from FS.

e.g

        at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487)
        at org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598)
        at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235)
        at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144)
        at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132)
        at org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126)
        at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
        at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)

This can be quite expensive in some FS like S3. Especially when table is partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can spend 1000s of seconds just waiting for these information to be pulled in.

Also, it would be good to remove FS.getContentSummary usage to find out file sizes.