Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12729

Suppress Avro Runtime Exceptions for Streaming

Details

    • Improvement
    • Status: Resolved
    • P3
    • Resolution: Fixed
    • None
    • None
    • io-java-avro
    • Patch

    Description

      The current design of ReadFileRangesFn continually throws any un-recoverable errors until the pipeline is turned down for streaming pipelines.

       

      These invalid file errors do not have a resolvable solution and should be logged as errors for the files in question to allow the pipeline to continue progressing.

       

      As a file which cannot be read will never recover via a dead letter queue design.  Since no recovery is possible we can simply log the errored file and continue processing.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              dhercher Dylan Hercher
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 2h
                  2h
                  Remaining:
                  Time Spent - 40m Remaining Estimate - 1h 20m
                  1h 20m
                  Logged:
                  Time Spent - 40m Remaining Estimate - 1h 20m
                  40m