Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36529

Decouple CPU with IO work in vectorized Parquet reader

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.3.0
    • None
    • SQL
    • None

    Description

      Currently it seems the vectorized Parquet reader does almost everything in a sequential manner:
      1. read the row group using file system API (perhaps from remote storage like S3)
      2. allocate buffers and store those row group bytes into them
      3. decompress the data pages
      4. in Spark, decode all the read columns one by one
      5. read the next row group and repeat from 1.

      A lot of improvements can be done to decouple the IO and CPU intensive work. In addition, we could parallelize the row group loading and column decoding, and utilizing all the cores available for a Spark task.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              csun Chao Sun
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated: