Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-12153

[Rust] [Parquet] Return file metadata after writing Parquet file

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0
    • Rust

    Description

      Parquet writers like delta-rs rely on the Parquet metadata to write file-level statistics for file pruning purposes.

      We currently do not expose these stats, requiring the writer to read the file that has just been written, to get the stats. This is more problematic for in-memory sinks, as there is currently no way of getting the metadata from the sink before it's persisted.

      Explore if we can expose these stats to the writer, to make the above easier.

      Attachments

        Issue Links

          Activity

            People

              nevi_me Neville Dipale
              nevi_me Neville Dipale
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m