Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22386 Data Source V2 improvements
  3. SPARK-29248

Pass in number of partitions to BuildWriter

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SQL
    • None

    Description

      When implementing a ScanBuilder, we require the implementor to provide the schema of the data and the number of partitions.

      However, when someone is implementing WriteBuilder we only pass them the schema, but not the number of partitions. This is an asymetrical developer experience. Passing in the number of partitions on the WriteBuilder would enable data sources to provision their write targets before starting to write. For example, it could be used to provision a Kafka topic with a specific number of partitions.

      Attachments

        Activity

          People

            edrevo Ximo Guanter
            edrevo Ximo Guanter
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: