Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.3.4, 3.1.1
-
None
-
None
Description
Consider the following query:
select col1 from tbl1 where year_partition=2019;
If the table has a lot of columns, currently we end up creating a TreeReader for each column, even when it won't pass the SearchArgument:
TreeReaderFactory.createTreeReader(TypeDescription, TreeReaderFactory$Context) line: 2339 TreeReaderFactory$StructTreeReader.<init>(int, TypeDescription, TreeReaderFactory$Context) line: 1974 TreeReaderFactory.createTreeReader(TypeDescription, TreeReaderFactory$Context) line: 2390 RecordReaderImpl(RecordReaderImpl).<init>(ReaderImpl, Reader$Options) line: 267 RecordReaderImpl.<init>(ReaderImpl, Reader$Options, Configuration) line: 67 ReaderImpl.rowsOptions(Reader$Options, Configuration) line: 83 OrcRawRecordMerger$OriginalReaderPairToRead.<init>(OrcRawRecordMerger$ReaderKey, Reader, int, RecordIdentifier, RecordIdentifier, Reader$Options, OrcRawRecordMerger$Options, Configuration, ValidWriteIdList, int) line: 446 OrcRawRecordMerger.<init>(Configuration, boolean, Reader, boolean, int, ValidWriteIdList, Reader$Options, Path[], OrcRawRecordMerger$Options) line: 1057 OrcInputFormat.getReader(InputSplit, Options) line: 2108 OrcInputFormat.getRecordReader(InputSplit, JobConf, Reporter) line: 2006 FetchOperator$FetchInputFormatSplit.getRecordReader(JobConf) line: 776
If the table has 1000 column, and spans N splits, we will end up creating 1000*N TreeReader objects when we might need only N (1/split).