Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-26871

TestCrudCompactorOnTez is flaky after HIVE-26479

    XMLWordPrintableJSON

Details

    Description

      The 3 tests in TestCrudCompactorOnTez which use the ProtoLoggingHook run at different times. Unfortunately, the 3 tests are run at the following times as described in the logs - 
      Test 1 - 

      INFO [main] compactor.TestCrudCompactorOnTez: Current time: 2022-12-15T23:57:44 

      Test 2 - 

      INFO [main] compactor.TestCrudCompactorOnTez: Current time: 2022-12-16T00:00:32 

      Test 3 - 

      INFO [main] compactor.TestCrudCompactorOnTez: Current time: 2022-12-16T00:04:12 

      As we can see, the tests are run on 2 different dates. Therefore, HiveProtoLoggingHook generates a unique event logs for every unique date. This is the behaviour of HiveProtoLoggingHook.
      https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L296-L310

      However the expectation from the test side, while generating the log readers is that there must be a single file in the log folder defined.
      https://github.com/apache/hive/blob/master/ql/src/test/org/apache/hadoop/hive/ql/hooks/TestHiveProtoLoggingHook.java#L310 

      Unfortunately, since there are 2 files which are generated (as mentioned in the logs as well), the following tests fail - 

      INFO [main] hooks.TestHiveProtoLoggingHook: List of paths: 
      INFO [main] hooks.TestHiveProtoLoggingHook: file:/home/jenkins/agent/workspace/internal-hive-flaky-check/itests/hive-unit/target/tmp/junit441259831997042392/junit3438435196942546140/date=2022-12-15
      INFO [main] hooks.TestHiveProtoLoggingHook: file:/home/jenkins/agent/workspace/internal-hive-flaky-check/itests/hive-unit/target/tmp/junit441259831997042392/junit3438435196942546140/date=2022-12-16 

      The solution is to make getTestReader() in TestHiveProtoLoggingHook more compatible with multiple event log file scenario and be able to generate multiple readers for all files present in the folder instead of fixating on a single file clause.

      Attachments

        Issue Links

          Activity

            People

              sbadhya Sourabh Badhya
              sbadhya Sourabh Badhya
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m