Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12095

spark_runner.py broken by Spark 3 upgrade.

Details

    • Bug
    • Status: Resolved
    • P1
    • Resolution: Fixed
    • None
    • 2.29.0
    • runner-spark
    • None

    Description

      The logic that finds the Spark job server jar has been broken by the Spark 3 upgrade, since it assumes the name of the jar corresponds directly to the Gradle target, which is no longer true.

      This path is still not documented properly, but some folks out there have already started using it (including TFX).

      Attachments

        Issue Links

          Activity

            People

              ibzib Kyle Weaver
              ibzib Kyle Weaver
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4h 50m
                  4h 50m