Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23539

Add support for Kafka headers in Structured Streaming

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 3.0.0
    • Structured Streaming
    • None

    Description

      Kafka headers were added in 0.11. We should expose them through our kafka data source in both batch and streaming queries. 

      This is currently blocked on version of Kafka in Spark from 0.10.1 to 1.0+ SPARK-18057

      Attachments

        Issue Links

          contains

          Documentation - Documentation or Website SPARK-28446 Document Kafka Headers support

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-18057 Update structured streaming kafka from 0.10.0.1 to 2.0.0

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #22282 [Github] Pull Request #22282 (dongjinleekr)

          Web Link GitHub Pull Request #22282

          Web Link GitHub Pull Request #25792

          Activity

            People

              dongjin Dongjin Lee
              tdas Tathagata Das
              Votes:
              4 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: