Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8145

Pubsub message size limit not taking size increase from base64 encoding into account

Details

    • Bug
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • io-java-gcp
    • None

    Description

      In the PubSubIO, the default max batch size is set to `10 * 1024 * 1024` bytes. This however does not take into account the size increase of base64 encoding the messages after the flush. Base64 encodes each set of three bytes into four bytes.

      Therefore the 'true' size limit placed on the unencoded batch should be
      x = ((10 * 1024 * 1024) / 4) * 3 = 7864320 bytes

      Attachments

        Activity

          People

            Unassigned Unassigned
            Primevenn Michael Yzvenn Wolanski
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 0.5h
                0.5h