Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.6.0
-
None
-
None
Description
When creating a convert for an array, Parquet Avro uses "array" as the field name name (see here) , but Parquet Hive SerDe uses "array_element" as the field name see here. In Spark SQL, our native Parquet support is following Parquet Avro's convention, for data generated by Parquet Hive SerDe, the array value cannot be correctly read and null will be returned.
Attachments
Issue Links
- is related to
-
SPARK-5508 Arrays and Maps stored with Hive Parquet Serde may not be able to read by the Parquet support in the Data Souce API
- Resolved