[HIVE-14828] Cloud/S3: Stats publishing should be on HDFS instead of S3 - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Minor
Resolution: Fixed
Affects Version/s: 1.2.0
Fix Version/s: 1.3.0
Component/s: Statistics
Labels:
None

Description

Currently, stats files are created in S3. Later as a part of FSStatsAggregator, it reads this file and populates MS again.

2016-09-23 05:57:46,772 INFO  [main]: fs.FSStatsPublisher (FSStatsPublisher.java:init(49)) - created : s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001
2016-09-23 05:57:46,773 DEBUG [main]: fs.FSStatsAggregator (FSStatsAggregator.java:connect(53)) - About to read stats from : s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001

Instead of this, stats can be written directly on to HDFS and read locally instead of S3, which would help in reducing couple of calls to S3.