Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-9629

JdbcIO seems to run out of connections in the connection pool and freezes pipeline

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • 2.18.0, 2.19.0, 2.21.0, 2.22.0, 2.23.0
    • 2.24.0
    • io-java-jdbc
    • Dataflow, Direct Runner on macOS Catalina.

    Description

      Greetings,

      I am using JdbcIO via the Scala wrappers provided in the Scio project. I am trying to read a few dozen tables in parallel from MySQL, but above 8 concurrent SELECT operations the pipeline freezes. With help of the Scio maintainers we've been able to isolate the issue as likely originating in JdbcIO running out of connections in the connection pool and idling indefinitely. The issue occurs both on the Direct Runner and Dataflow.

      Please see linked issue for more context: https://github.com/spotify/scio/issues/2774

      Attachments

        Issue Links

          Activity

            People

              lcwik Luke Cwik
              muscovite bob Boris Shilov
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h