Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21375

Closing TransactionBatch closes FileSystem for other batches in Hive streaming v1

    XMLWordPrintableJSON

Details

    Description

      The patch in HIVE-13151 added FileSystem.closeAllForUGI(ugi); to the close method of HiveEndPoint for the legacy Streaming API. This seems to have a side effect of closing the FileSystem for all open TransactionBatches as used by NiFi and Storm when writing to multiple partitions. Setting fs.hdfs.impl.disable.cache=true negates the issue but at a performance cost.

      Attachments

        Issue Links

          Activity

            People

              szita Ádám Szita
              Absolutesantaja Shawn Weeks
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h