Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-26609

Cleanup external table directories created in qtests after test run

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Tests
    • None

    Description

      In many qtests we are creating external tables by setting explicitly the location of the table.

      https://github.com/apache/hive/blob/566f48d3d3fc740ef958bdf963e511e0853da402/ql/src/test/queries/clientnegative/authorization_uri_create_table_ext.q#L7

      If the test does not remove explicitly the directory (as it happens above) then it remains there and may cause flakiness and unrelated test failures if other tests happen to use the same directory somehow.

      A recent case where this problem appeared (directory conflict between tests) is logged under HIVE-26584. There the solution was to add explicit rm commands in the qfiles.

      A more general solution would be to handle the cleanup of such directories inside QTestUtil.clearTablesCreatedDuringTests.

      The idea is to get the location for an external table from the metastore and remove the respective directory if it is under a known "safe" directory such as ${system:test.tmp.dir}.

      As discussed under HIVE-26584 it might be risky to forcefully delete any kind of directory coming from an external table at the risk of corrupting the development environment. If we restrict the cleanup to known directories it should be fine though.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              zabetak Stamatis Zampetakis
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: