Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
The Beam portability layer is being developed on Spark and Flink runners. Beam portability layer enables for programming in Java, Go, and Python, and enables running Beam programs programmed with the different languages on Nemo. This will be great for attracting more users to Apache Nemo. Related link: https://beam.apache.org/roadmap/portability/
- Watch https://www.youtube.com/watch?v=I2ZqWAbbjUk for an overview and to get an idea
- Refer to files under /runners/portability package, check requirements for the PortableRunner.java class
- Flink is the system that provides most functionalities regarding portability, we should refer to PortableExecutionTest class, as well as FlinkPortableClientEntryPoint, FlinkPortablePipelineTranslator (for both batch and streaming), FlinkPortableRunnerResult classes, under /runners/flink, which implement the portability layer.
- Spark runner also has points that we could refer to: PortableBatchMode,
- We will need to make our `Nemo PortablePipelineOptions , as well as ClientEntryPoint, PipelineTranslator and PortableRunnerResult, just
- ClientEntryPoint, PipelineTranslator and PortableRunnerResult, just
- PipelineTranslator and PortableRunnerResult, just
- PortableRunnerResult, just