Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-2891 Spark runs portable pipelines
  3. BEAM-2590

SparkRunner shim for Job API

    XMLWordPrintableJSON

Details

    Description

      Whatever the result of https://s.apache.org/beam-job-api we will need a way for the JVM-based SparkRunner to receive and run pipelines authors in Python.

      Attachments

        Issue Links

          is blocked by

          Task - A task that needs to be done. BEAM-6685 JobServer driver with stubbed out spark related component

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Task - A task that needs to be done. BEAM-6686 Spark job invoker with stubbed out translator

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Task - A task that needs to be done. BEAM-6687 Spark Translator - GROUP_BY_KEY_TRANSFORM_URN

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Task - A task that needs to be done. BEAM-6688 Spark Translator - FLATTEN_TRANSFORM_URN

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Task - A task that needs to be done. BEAM-6689 Spark Translator - IMPULSE_TRANSFORM_URN

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Task - A task that needs to be done. BEAM-6690 Spark Translator - ASSIGN_WINDOWS_TRANSFORM_URN

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Task - A task that needs to be done. BEAM-6691 Spark Translator - ExecutableStage.URN

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Task - A task that needs to be done. BEAM-6692 Spark Translator - RESHUFFLE_URN

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-7037 Spark portable runner: run wordcount.py

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          links to

          Web Link GitHub Pull Request #8115

          Activity

            People

              ibzib Kyle Weaver
              kenn Kenneth Knowles
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 40m
                  2h 40m