Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14546

Allow Partitioner to return -1 to indicate default partitioning

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.3.1
    • None
    • producer
    • None

    Description

      Prior to KIP-794 it was possible to create a custom Partitioner that could delegate to the DefaultPartitioner.  DefaultPartitioner has been deprecated so we can now only delegate to BuiltInPartitioner.partitionForKey which does not handle a non-keyed message.  Hence there is now no mechanism for a custom Partitioner to fallback to default partitioning, e.g. for the non-keyed sticky case.

      I would like to propose that KafkaProducer.partition(...) not throw IllegalArgumentException if the Partitioner returns RecordMetadata.UNKNOWN_PARTITION and instead continue with the default behaviour.  Maybe with a configuration flag to enable this behaviour so as not to break existing expectations?

      Why was Partitioner delegation with default fallback useful?

      1. A single Producer can be used to write to multiple Topics where each Topic may have different partitioning requirements.  The Producer can only have a single Partitioner so the Partitioner needs to be able to switch behaviour based on the Topic, including the need to fallback to default behaviour if a given Topic does not have a custom requirement.
      2. Multiple services may need to produce to the same Topic and these services may be authored by different teams.  A single custom Partitioner that encapsulates all Topic specific partitioning logic can be used by all teams at all times for all Topics ensuring that mistakes are not made.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              jim_b_o James Olsen
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: