Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11685

Restarting Metastore kills Compactions - store Hadoop job id in COMPACTION_QUEUE

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.0.1
    • None
    • Metastore, Transactions
    • None

    Description

      CompactorMR submits MR job to do compaction and waits for completion.
      If the metastore need to be restarted, it will kill in-flight compactions.

      I ideally we'd want to add job ID to the COMPACTION_QUEUE table (and include that in SHOW COMPACTIONS) and poll for it or register a callback so that the job survives Metastore restart

      Also,
      when running revokeTimedoutWorker() make sure to use this JobId to kill the job is it's still running.
      Alternatively, if it's still running, maybe just a assign a new worker_id and let it continue to run.

      Attachments

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue HIVE-12832 RDBMS schema changes for HIVE-11388

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-11388 Allow ACID Compactor components to run in multiple metastores

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-15337 Enhance Show Compactions output with JobId and start time for "attempted" state

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: