Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21052

Make sure transactions get cleaned if they are aborted before addPartitions is called

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.0.0, 3.1.1
    • None
    • Transactions

    Description

      If the transaction is aborted between openTxn and addPartitions and data has been written on the table the transaction manager will think it's an empty transaction and no cleaning will be done.
      This is currently an issue in the streaming API and in micromanaged tables. As proposed by ekoifman this can be solved by:

      • Writing an entry with a special marker to TXN_COMPONENTS at openTxn and when addPartitions is called remove this entry from TXN_COMPONENTS and add the corresponding partition entry to TXN_COMPONENTS.
      • If the cleaner finds and entry with a special marker in TXN_COMPONENTS that specifies that a transaction was opened and it was aborted it must generate jobs for the worker for every possible partition available.

      cc ewohlstadter

      Attachments

        1. Aborted Txn w_Direct Write.pdf
          67 kB
          Eugene Koifman
        2. HIVE-21052.1.patch
          738 kB
          Jaume M
        3. HIVE-21052.10.patch
          94 kB
          Jaume M
        4. HIVE-21052.11.patch
          94 kB
          Jaume M
        5. HIVE-21052.12.patch
          94 kB
          Jaume M
        6. HIVE-21052.2.patch
          738 kB
          Jaume M
        7. HIVE-21052.3.patch
          198 kB
          Jaume M
        8. HIVE-21052.4.patch
          727 kB
          Jaume M
        9. HIVE-21052.5.patch
          727 kB
          Jaume M
        10. HIVE-21052.6.patch
          731 kB
          Jaume M
        11. HIVE-21052.7.patch
          731 kB
          Jaume M
        12. HIVE-21052.8.patch
          82 kB
          Jaume M
        13. HIVE-21052.9.patch
          87 kB
          Jaume M

        Issue Links

          causes

          Task - A task that needs to be done. HIVE-25502 Cleaner causes data loss when processing aborted txn with DP

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-24374 Do not remove metadata for aborted writes that were batched with valid ones in the same delta file

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-21150 Don't block Cleaner at the end of the loop

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-21118 Make sure if dynamic partitions is true only there's only one writeId allocated

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          supercedes

          Bug - A problem which impairs or prevents the functions of the product. HIVE-20435 Failed Dynamic Partition Insert into insert only table may loose transaction metadata

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          links to

          Web Link GitHub Pull Request #1415

          Web Link GitHub Pull Request #1548

          Web Link Review Board

          Activity

            People

              jmarhuen Jaume M
              jmarhuen Jaume M
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 12.5h
                  12.5h