Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-25970

Missing messages in HS2 operation logs

    XMLWordPrintableJSON

Details

    Description

      After HIVE-22753 & HIVE-24590, with some unlucky timing of events, operation log messages can get lost and never appear in the appropriate files.

      The changes in HIVE-22753 will prevent a HushableRandomAccessFileAppender from being created if the latter refers to a file that has been closed in the last second. Preventing the creation of the appender also means that the message which triggered the creation will be lost forever. In fact any message (for the same query) that comes in the interval of 1 second will be lost forever.

      Before HIVE-24590 the appender/file was closed only once (explicitly by HS2) and thus the problem may be very hard to notice in practice. However, with the arrival of HIVE-24590 appenders may close much more frequently (and not via HS2) making the issue reproducible rather easily. It suffices to set hive.server2.operation.log.purgePolicy.timeToLive property very low and check the operation logs.

      The problem was discovered by investigating some intermittent failures in operation logging tests (e.g., TestOperationLoggingAPIWithTez).

      Attachments

        Issue Links

          Activity

            People

              zabetak Stamatis Zampetakis
              zabetak Stamatis Zampetakis
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m