Description
Parquet 1.11 new features:
PARQUET-1201 - Column indexes
PARQUET-1253 - Support for new logical type representation
PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
Parquet 1.11 introduced the following configurations:
| Property name | Default value |
|---|---|
| parquet.columnindex.truncate.length | 64 |
| parquet.statistics.truncate.length | Integer.MAX_VALUE |
| parquet.page.row.count.limit | 20000 |
| parquet.page.write-checksum.enabled | true |
| parquet.filter.columnindex.enabled | true |
| parquet.page.verify-checksum.enabled | false |
Attachments
Issue Links
- blocks
-
SPARK-26797 Start using the new logical types API of Parquet 1.11.0 instead of the deprecated one
-
- In Progress
-
- is blocked by
-
SPARK-27733 Upgrade to Avro 1.10.1
-
- Resolved
-
-
PARQUET-1739 Make Spark SQL support Column indexes
-
- Resolved
-
-
PARQUET-1434 Release parquet-mr 1.11.0
-
- Resolved
-
- links to