Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19416

Create single version transactional table metastore statistics for aggregation queries

    XMLWordPrintableJSON

Details

    Description

      This adds accurate stats support to Hive transactional (insert-only and full ACID) tables, so that some queries from these tables can be answered from stats and also so that the stats could be used for more optimization. This support can be enabled via a config flag, and is on by default.

      This is achieved via the following changes, that basically start us on the path of treating ACID stats the same way we treat ACID data:
      In addition to existing JSON blob, we store a write ID of the latest stats writer with each table and partition. Any writer updating the stats or altering the stats state for a txn table has to record his write ID - if the write ID and some other context is not provided by the caller of alter, the alter table/partition operation fails. It's the responsibility of the writer to not commit its transaction if the operation fails.
      In future, we'd like to move the stats state into actual stats tables, but for now it's (logically) colocated with the existing json parameter.

      In addition to its write ID, most callers (with the exception of the ones that cannot have races, e.g. create table) provide their own txn state (write ID list) for the table. The existing stats' write ID is verified against this state. If the write ID is not visible to the updater, we still update the stats, but set the stats state to invalid; that basically means that two parallel operations that cannot see each others' data output are updating the stats.

      This way, txn stats stay valid as the result of a sequence of non-conflicting updates that can all see each other and account for each others' data for the stats. Any parallel updates invalidate the stats.
      This is necessary because unlike data, stats are a single version summary of the table. To be able to support parallel operations with valid stats, each stats update would need to write a separate record, and would also need to write mergeable records that only reflect its own changes, instead of the final view of the table stats (two of those are hard or impossible to merge).

      This approach resulted in a few changes to alter/etc APIs in metastore; it also requires that many alter operations, as well as analyze table, allocate a write ID (because they affect stats-that-are-treated-like-data, and so are essentially a write operation).

      The reader, in turn, verifies that the stats are valid and written by a write ID that is itself valid given the reader's transactional state (i.e. not aborted, nor in progress). This is done on metastore side; if the stats are invalid for the reader, we transparently update the stats state returned to the caller to mark the stats as inaccurate.

      We've considered (and actually implemented) an alternative approach of recording the full txn state of the stats writer to be compared with the state of the stats reader (to see if they are compatible and avoid the extra write IDs and strict write-time checks), however it results in problems with partitioned tables, where not all writes affect all partitions, and so the stats state of all the untouched partitions becomes invalid once a subset of partitions is updated (because we cannot tell whether the write ID, a table level operation, didn't touch the partition, or did touch it but didn't record the stats). Additionally, storing full txn state for every partition and table can be expensive, especially in extreme cases where the watermark doesn't advance for a while for some reason.

      Attachments

        Issue Links

          1.
          Modify metastore to have/access persistent tables for stats Sub-task Closed Steve Yeom
          2.
          Resolve explain.out differences introduced by transactional metastore statistics Sub-task Closed Unassigned
          3.
          Modify Hive to support transactional-stats-using aggregation queries with all other than COUNT Sub-task Resolved Steve Yeom
          4.
          Modify metastore to have application logic to retrieve/update transactional table stats Sub-task Closed Steve Yeom
          5.
          master-txnstats branch - make sure SQL changes are in correct upgrade scripts Sub-task Closed Steve Yeom
          6.
          master-txnstats branch - don't get write IDs from metastore when it's not safe Sub-task Resolved Sergey Shelukhin
          7.
          merge master-txnstats branch Sub-task Closed Sergey Shelukhin
          8.
          handle concurrent INSERTS Sub-task Closed Steve Yeom
          9.
          support txn stats in CachedStore Sub-task Closed Sergey Shelukhin
          10.
          stats_nonpart.q test run shows possibly wrong results and other bugfixes Sub-task Closed Steve Yeom
          11.
          INSERT INTO SELECT case test may be needed. Sub-task Resolved Unassigned
          12.
          Verify ACID table UPDATE/DELETE commands on transactional stats. Sub-task Closed Sergey Shelukhin
          13.
          Verify any other aggregation functions other than COUNT Sub-task Resolved Unassigned
          14.
          add ACID stats support to background stats updater and fix bunch of edge cases found in SU tests Sub-task Closed Sergey Shelukhin
          15.
          query9.q fails Sub-task Closed Steve Yeom
          16.
          TestTxnCommands2#testNonAcidToAcidConversion1 fails Sub-task Closed Steve Yeom
          17.
          metadata_only_queries.q fails Sub-task Closed Steve Yeom
          18.
          Checking writeIdList per table may not check the commit level of a partition on a partitioned table Sub-task Closed Sergey Shelukhin
          19.
          acid_table_stats, acid_no_buckets, etc - query result change on the branch Sub-task Closed Steve Yeom
          20.
          fix TestReplicationScenarios on the branch Sub-task Closed Sergey Shelukhin
          21.
          add parallel insert, analyze, iow tests Sub-task Closed Sergey Shelukhin
          22.
          remove NUM_FILES check Sub-task Closed Sergey Shelukhin
          23.
          remove txnID argument for txn stats methods Sub-task Closed Sergey Shelukhin
          24.
          add a config flag to turn off txn stats Sub-task Closed Sergey Shelukhin
          25.
          remove EnvironmentContext usage and add proper request APIs Sub-task Closed Sergey Shelukhin
          26.
          stats_part2.q fails Sub-task Closed Unassigned
          27.
          Bypass HMS CachedStore for transactional stats Sub-task Closed Steve Yeom
          28.
          Acid tables should not use footer scan for analyze Sub-task Closed Sergey Shelukhin
          29.
          verify that analyze and analyze for columns manage txn stats state correctly Sub-task Closed Sergey Shelukhin

          Activity

            People

              steveyeom2017 Steve Yeom
              steveyeom2017 Steve Yeom
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: