Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12631

Allow Inheritance from KafkaUnboundedSource

Details

    • New Feature
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • io-java-kafka
    • None

    Description

      Reason:

      We have dataflow to call one internal API (the API call take 60ms). So we want to start more threads to call this API. Kafka IO thread is control by split by default. And split is control by total partitions and system suggestion number. And system suggestion number is 4 * max workers. which is too low for us.  

      Solution:

      Use facture to allow user inheritance KafkaUnboundedSource.  and override the split function like

       

      // code placeholder
          @Override
          public List<KafkaUnboundedSource<K, V>> split(int desiredNumSplits, PipelineOptions options)
                  throws Exception {
              int split = this.getMaxSplits();
              if(split <= 0){
                  return super.split(desiredNumSplits, options);
              }else{
                  return super.split(split, options);
              }
          }
      
      

       this may also useful for Override other feature of the KafkaUnboundedSource without copy all the io code.

      Attachments

        Activity

          People

            Unassigned Unassigned
            renhongxiang HongXiang Ren
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 4h
                4h
                Remaining:
                Remaining Estimate - 4h
                4h
                Logged:
                Time Spent - Not Specified
                Not Specified