Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-23883

Streaming does not flush the side file

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Streaming, Transactions
    • None

    Description

      When a streaming write commits a mid-batch write with connection.commitTransaction() then it tries to flush the sideFile with OrcInputFormat.SHIMS.hflush(flushLengths). This uses FSOutputSummer.flush, which does not flush the buffer data to the disk so the actual data is not written.

      Had to remove the check from the end of the streaming tests in TestCrudCompactorOnTez.java

            CompactorTestUtilities.checkAcidVersion(fs.listFiles(new Path(table.getSd().getLocation()), true), fs,
                conf.getBoolVar(HiveConf.ConfVars.HIVE_WRITE_ACID_VERSION_FILE),
                new String[] { AcidUtils.DELTA_PREFIX });
      

      These checks verifies the _flush_length files, and they would fail otherwise.

      Attachments

        Activity

          People

            Unassigned Unassigned
            pvary Peter Vary
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: