Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14828

Cloud/S3: Stats publishing should be on HDFS instead of S3

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.2.0
    • 1.3.0
    • Statistics
    • None

    Description

      Currently, stats files are created in S3. Later as a part of FSStatsAggregator, it reads this file and populates MS again.

      2016-09-23 05:57:46,772 INFO  [main]: fs.FSStatsPublisher (FSStatsPublisher.java:init(49)) - created : s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001
2016-09-23 05:57:46,773 DEBUG [main]: fs.FSStatsAggregator (FSStatsAggregator.java:connect(53)) - About to read stats from : s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001

      Instead of this, stats can be written directly on to HDFS and read locally instead of S3, which would help in reducing couple of calls to S3.

      Attachments

        1. HIVE-14828.branch-2.0.001.patch
          1 kB
          Rajesh Balamohan
        2. HIVE-14828.branch-1.2.001.patch
          2 kB
          Rajesh Balamohan
        3. HIVE-14828.1.patch
          1 kB
          Rajesh Balamohan

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-13925 ETL optimizations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              rajesh.balamohan Rajesh Balamohan
              rajesh.balamohan Rajesh Balamohan
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: