Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31255

DataSourceV2: Add metadata columns

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Done
    • 3.0.0
    • 3.1.0
    • SQL
    • None

    Description

      DSv2 should support reading additional metadata columns that are not in a table's schema. This allows users to project metadata like Kafka's offset, timestamp, and partition. It also allows other sources to expose metadata like file and row position.

      Attachments

        Issue Links

          causes

          Sub-task - The sub-task of the issue SPARK-33524 Change `InMemoryTable` not to use Tuple.hashCode for `BucketTransform`

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #30415 [Github] Pull Request #30415 (rdblue)

          Web Link GitHub Pull Request #28027

          Activity

            People

              rdblue Ryan Blue
              rdblue Ryan Blue
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: