Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-442

Beam portability layer support for Apache Nemo

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None

    Description

      The Beam portability layer is being developed on Spark and Flink runners. Beam portability layer enables for programming in Java, Go, and Python, and enables running Beam programs programmed with the different languages on Nemo. This will be great for attracting more users to Apache Nemo. Related link: https://beam.apache.org/roadmap/portability/

      • Watch https://www.youtube.com/watch?v=I2ZqWAbbjUk for an overview and to get an idea
      • Refer to files under /runners/portability package, check requirements for the PortableRunner.java class
      • Flink is the system that provides most functionalities regarding portability, we should refer to PortableExecutionTest class, as well as FlinkPortableClientEntryPoint, FlinkPortablePipelineTranslator (for both batch and streaming), FlinkPortableRunnerResult classes, under /runners/flink, which implement the portability layer.
      • Spark runner also has points that we could refer to: PortableBatchMode,
      • We will need to make our `Nemo PortablePipelineOptions , as well as ClientEntryPoint, PipelineTranslator and PortableRunnerResult, just
      • ClientEntryPoint, PipelineTranslator and PortableRunnerResult, just
      • PipelineTranslator and PortableRunnerResult, just
      • PortableRunnerResult, just

      Attachments

        Activity

          People

            Unassigned Unassigned
            wonook Wonook
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: