Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11685

Restarting Metastore kills Compactions - store Hadoop job id in COMPACTION_QUEUE

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.0.1
    • None
    • Metastore, Transactions
    • None

    Description

      CompactorMR submits MR job to do compaction and waits for completion.
      If the metastore need to be restarted, it will kill in-flight compactions.

      I ideally we'd want to add job ID to the COMPACTION_QUEUE table (and include that in SHOW COMPACTIONS) and poll for it or register a callback so that the job survives Metastore restart

      Also,
      when running revokeTimedoutWorker() make sure to use this JobId to kill the job is it's still running.
      Alternatively, if it's still running, maybe just a assign a new worker_id and let it continue to run.

      Attachments

        Issue Links

          There are no Sub-Tasks for this issue.

          Activity

            People

              Unassigned Unassigned
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: