Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-9660

StreamingDataflowWorker has confusing exception on commits over 2GB

Details

    • Bug
    • Status: Resolved
    • P3
    • Resolution: Fixed
    • 2.18.0, 2.19.0
    • 2.21.0
    • runner-dataflow
    • None

    Description

      Commits over 2GB have a negative serialized commit size.
      When not using streaming engine the max commit limit is 2GB.
      https://github.com/apache/beam/blob/v2.19.0/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java#L450
      There appears to be a logging regression introduced by
      https://github.com/apache/beam/pull/10013

      With the new code, if the serialization overflows the estimated bytes is set to Integer.MAX which equals the commit limit for appliance.
      Then the comparison here:
      https://github.com/apache/beam/blob/v2.19.0/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java#L1371
      which uses > does not trigger and the large commit is just passed on to the commit queue, triggering the exception seen in #3 [2] when the weigher uses the negative serialized size for the semaphore acquire call.

      So previously where we would have thrown a KeyCommitTooLargeException we are throwing the IllegalArgumentException.
      From that exception description: https://github.com/apache/beam/blob/v2.19.0/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java#L236
      ". This may be caused by grouping a very "
      + "large amount of data in a single window without using Combine,"
      + " or by producing a large amount of data from a single input element."

      The overflow could be remembered explicitly instead of just comparing with max.

      Attachments

        Issue Links

          Activity

            People

              scwhittle Sam Whittle
              scwhittle Sam Whittle
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h