Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.1.0
-
None
-
None
Description
I create a ACID table with ORC format:
CREATE TABLE `some.TableA`( .... ) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' TBLPROPERTIES ( 'bucketing_version'='2', 'orc.compress'='SNAPPY', 'transactional'='true', 'transactional_properties'='default')
After executing merge into operation:
MERGE INTO some.TableA AS a USING (SELECT vend_no FROM some.TableB UNION ALL SELECT vend_no FROM some.TableC) AS b ON a.vend_no=b.vend_no WHEN MATCHED THEN DELETE
the problem happend(when selecting the TableA, the exception happens too):
java.io.IOException: java.io.IOException: Two readers for {originalWriteId: 4, bucket: 536870912(1.0.0), row: 2434, currentWriteId 25}: new [key={originalWriteId: 4, bucket: 536870912(1.0.0), row: 2434, currentWriteId 25}, nextRecord={2, 4, 536870912, 2434, 25, null}, reader=Hive ORC Reader(hdfs://hdpprod/warehouse/tablespace/managed/hive/some.db/tableA/delete_delta_0000015_0000026/bucket_00001, 9223372036854775807)], old [key={originalWriteId: 4, bucket: 536870912(1.0.0), row: 2434, currentWriteId 25}, nextRecord={2, 4, 536870912, 2434, 25, null}, reader=Hive ORC Reader(hdfs://hdpprod/warehouse/tablespace/managed/hive/some.db/tableA/delete_delta_0000015_0000026/bucket_00000
Through orc_tools I scan all the files(bucket_00000,bucket_00001,bucket_00002) under delete_delta and find all rows of files are the same.I think this will cause the same key(RecordIdentifer) when scan the bucket_00001 after bucket_00000 but I don't know why all the rows are the same in these bucket files.
Attachments
Attachments
Issue Links
- is caused by
-
HIVE-16812 VectorizedOrcAcidRowBatchReader doesn't filter delete events
-
- Closed
-