Uploaded image for project: 'Apache Avro'
  1. Apache Avro
  2. AVRO-3768

Provide InputStream implementation that wraps an Iterator of Record

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • java
    • None

    Description

      Hello,

      I have some code that generates avro record (using GenericData.Record precisely) record per record (I used an Iterator<GenericData.Record> in practice, but we could imagine anything similar including an actual java.util.Stream or even an avro-provided interface), and I would like to serialized it to some external system (an http request in my particular case). So basically the data is generated as it is pulled.

      Right now, the only option I found is to use a combination of java.io.PipedInputStream and PipedOutputStream, wrapping the later inside a DataFileWriter and then feeding the records to the writer (in a separate thread) so that the PipedInputStream can be read by anything else.

      As you can see this is a bit cumbersome and a more straightforward approach would be welcomed. I tried to implement this myself but I admit I got lost in all the moving pieces and I couldn't find a simple way of doing that without implementing a lot of low-level stuff. So maybe I'm missing something or we could add something to avro

      Attachments

        1. GenericDataIteratorInputStream.java
          2 kB
          Christophe Le Saec
        2. IteratorInputStream.java
          1 kB
          Christophe Le Saec
        3. Main.java
          1 kB
          Christophe Le Saec

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            victor.noel Victor

            Dates

              Created:
              Updated:

              Slack

                Issue deployment