Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23541

Allow Kafka source to read data with greater parallelism than the number of topic-partitions

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.4.0
    • Structured Streaming
    • None

    Description

      Currently, when the Kafka source reads from Kafka, it generates as many tasks as the number of partitions in the topic(s) to be read. In some case, it may be beneficial to read the data with greater parallelism, that is, with more number partitions/tasks. That means, offset ranges must be divided up into smaller ranges such the number of records in partition ~= total records in batch / desired partitions. This would also balance out any data skews between topic-partitions.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. SPARK-28489 KafkaOffsetRangeCalculator.getRanges may drop offsets

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-29799 Split a kafka partition into multiple KafkaRDD partitions in the kafka external plugin for Spark Streaming

          • Major - Major loss of function.
          • Resolved
          is related to

          Documentation - Documentation or Website SPARK-28464 Document kafka minPartitions option in "Structured Streaming + Kafka Integration Guide"

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Pull request #20698 [Github] Pull Request #20698 (tdas)

          Web Link GitHub Pull Request #20698

          Activity

            People

              tdas Tathagata Das
              tdas Tathagata Das
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: