Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-17959

[C++][Dataset] Optimize Parquet column projection for subset of nested field

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • C++

    Description

      Currently, when reading a subfield of a nested column of a Parquet file using the Dataset API, we read the full parent column instead of only the requested field. This should be optimized to only read the field itself.

      This was left as a TODO in ARROW-14658 (https://github.com/apache/arrow/pull/11704) which added the initial support for nested field refs in dataset scanning (https://github.com/apache/arrow/blob/c29ca51f44eaf41c3a2f6f72e3e23a7b428211c2/cpp/src/arrow/dataset/file_parquet.cc#L240-L246):

        if (field) {
          // TODO(ARROW-1888): support fine-grained column projection. We should be
          // able to materialize only the child fields requested, and not the entire
          // top-level field.
          // Right now, if enabled, projection/filtering will fail when they cast the
          // physical schema to the dataset schema.
          AddColumnIndices(*toplevel, columns_selection);
      

      Some relevant comments at https://github.com/apache/arrow/pull/11704#discussion_r749733765. ARROW-1888 was mentioned as a blocker back then, but this is resolved in the meantime.

      Attachments

        Activity

          People

            Unassigned Unassigned
            jorisvandenbossche Joris Van den Bossche
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: