Uploaded image for project: 'Apache Avro'
  1. Apache Avro
  2. AVRO-3266

Output stream incompatible with MagicS3GuardCommitter

    XMLWordPrintableJSON

Details

    Description

      Avro's output stream can not be used in combination with the MagicS3GuardCommitter

      Error: java.lang.ClassCastException: class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

      The reason for this problem is that AvroOutputFormatBase.getAvroFileOutputStream tries to cast the outputcommiter to a FileOutputCommitter.

      It can be solved by casting to a PathOutputCommitter instead (which is a superclass of both the FileOutputCommitter and the MagicS3GuardCommitter)

      Attachments

        Issue Links

          Activity

            People

              eejbyfeldt Emil Ejbyfeldt
              mjwiq Michiel de Jong
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4h 50m
                  4h 50m