Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7120

Turn `WindowedValue<T>` into `T` for all the interfaces of Data Plane

Details

    Description

      The main purpose of this JIRA is Turn `WindowedValue<T>` into `T` for all the interfaces of Data Plane.

      The interface definitions for sending and receiving data in the code as follows:

      • org.apache.beam.runners.fnexecution.data.FnDataService
        public interface FnDataService {
          <T> InboundDataClient receive(LogicalEndpoint inputLocation, Coder<WindowedValue<T>> coder, FnDataReceiver<WindowedValue<T>> listener);
          <T> CloseableFnDataReceiver<WindowedValue<T>> send(
              LogicalEndpoint outputLocation, Coder<WindowedValue<T>> coder);
        }
        
      • org.apache.beam.fn.harness.data.BeamFnDataClient
        public interface BeamFnDataClient {
          <T> InboundDataClient receive(ApiServiceDescriptor apiServiceDescriptor, LogicalEndpoint inputLocation, Coder<WindowedValue<T>> coder, FnDataReceiver<WindowedValue<T>> receiver);
          <T> CloseableFnDataReceiver<WindowedValue<T>> send(BeamFnDataGrpcClient Endpoints.ApiServiceDescriptor apiServiceDescriptor, LogicalEndpoint outputLocation, Coder<WindowedValue<T>> coder);
        }
        

      Both `Coder<WindowedValue<T>>` and `FnDataReceiver<WindowedValue<T>>` use `WindowedValue` as the data structure that both sides of Runner and SDK Harness know each other. Control Plane/Data Plane/State Plane/Logging is a highly abstraction, such as Control Plane and Logging, these are common requirements for all multi-language platforms. For example, the Flink community is also discussing how to support Python UDF, as well as how to deal with docker environment. how to data transfer, how to state access, how to logging etc. If Beam can further abstract these service interfaces, i.e., interface definitions are compatible with multiple engines, and finally provided to other projects in the form of class libraries, it definitely will help other platforms that want to support multiple languages. So we can further abstract the interface definition of FnDataService's BeamFnDataClient, take the FnDataService#receive interface as an example, and turn `WindowedValue<T>` into `T` so that other platforms can be extended arbitrarily, as follows:

      <T> InboundDataClient receive(LogicalEndpoint inputLocation, Coder<T> coder, FnDataReceiver<T>> listener);
      

      For details, see the mail discussion list:

      https://lists.apache.org/list.html?dev@beam.apache.org:lte=1M:%5BDISCUSS%5D%20Turn%20%60WindowedValue

      Attachments

        Activity

          People

            Unassigned Unassigned
            sunjincheng121 sunjincheng
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: