Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17657

SequenceFile.Writer should implement StreamCapabilities

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Following exception is thrown whenever we invoke ProtoMessageWriter.hflush on S3 from Tez, which internally calls org.apache.hadoop.io.SequenceFile$Writer.hflush ->  org.apache.hadoop.fs.FS DataOutputStream.hflush -> S3ABlockOutputStream.hflush which is not implemented and throws java.lang.UnsupportedOperationException. 

      bdffe22d96ae [mdc@18060 class="yarn.YarnUncaughtExceptionHandler" level="ERROR" thread="HistoryEventHandlingThread"] Thread Thread[HistoryEventHandlingThread, 5,main] threw an Exception.^Mjava.lang.UnsupportedOperationException: S3A streams are not Syncable^M at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.hflush(S3ABlockOutputStream.java:657)^M at org.apache.hadoop.fs.FS DataOutputStream.hflush(FSDataOutputStream.java:136)^M at org.apache.hadoop.io.SequenceFile$Writer.hflush(SequenceFile.java:1367)^M at org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(ProtoMessageWr iter.java:64)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.finishCurrentDag(ProtoHistoryLoggingService.java:239)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.han dleEvent(ProtoHistoryLoggingService.java:198)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.loop(ProtoHistoryLoggingService.java:153)^M at java.lang.Thread.run(Thread.java:748)^M

      In order to fix this issue we should implement StreamCapabilities in SequenceFile.Writer. Also, we should fall back to flush(), if hflush() is not supported. 

      Attachments

        Issue Links

          Activity

            People

              kishendas Kishen Das
              kishendas Kishen Das
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3h 50m
                  3h 50m