Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11388

Allow ACID Compactor components to run in multiple metastores

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 1.0.0
    • 1.3.0, 2.1.0
    • Transactions
    • None

    Description

      (this description is no loner accurate; see further comments)

      org.apache.hadoop.hive.ql.txn.compactor.Initiator is a thread that runs inside the metastore service to manage compactions of ACID tables. There should be exactly 1 instance of this thread (even with multiple Thrift services).

      This is documented in https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration but not enforced.

      Should add enforcement, since more than 1 Initiator could cause concurrent attempts to compact the same table/partition - which will not work.

      Attachments

        1. HIVE-11388.patch
          26 kB
          Eugene Koifman
        2. HIVE-11388.2.patch
          26 kB
          Eugene Koifman
        3. HIVE-11388.4.patch
          37 kB
          Eugene Koifman
        4. HIVE-11388.5.patch
          36 kB
          Eugene Koifman
        5. HIVE-11388.6.patch
          38 kB
          Eugene Koifman
        6. HIVE-11388.7.patch
          39 kB
          Eugene Koifman
        7. HIVE-11388.branch-1.patch
          40 kB
          Eugene Koifman

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. HIVE-13013 Further Improve concurrency in TxnHandler

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HIVE-13344 port HIVE-12902 to 1.x line

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-11685 Restarting Metastore kills Compactions - store Hadoop job id in COMPACTION_QUEUE

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-12725 CompactionTxnHandler.findNextToCompact() may produce "Operation not allowed after ResultSet closed"

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-12592 Expose connection pool tuning props in TxnHandler

          • Major - Major loss of function.
          • Open

          Activity

            People

              ekoifman Eugene Koifman
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: