Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.10.0
-
None
-
None
Description
When bufferSizeProp is null, CodecUtils.getDecompressedInputStreamWithBufferSize runs into NullPointerException:
// code placeholder java.io.IOException: java.lang.NullPointerException at org.apache.tez.runtime.library.common.shuffle.ShuffleUtils.shuffleToMemory(ShuffleUtils.java:142) at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyMapOutput(FetcherOrderedGrouped.java:550) at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:283) at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.fetchNext(FetcherOrderedGrouped.java:182) at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:194) at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:57) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at java.util.Hashtable.get(Hashtable.java:363) at java.util.Properties.getProperty(Properties.java:969) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200) at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1254) at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1479) at org.apache.tez.runtime.library.utils.CodecUtils.getDecompressedInputStreamWithBufferSize(CodecUtils.java:81) at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readToMemory(IFile.java:824) at org.apache.tez.runtime.library.common.shuffle.ShuffleUtils.shuffleToMemory(ShuffleUtils.java:121) ... 12 more
Configuration.getInt(String name, int defaultValue) assumes theĀ name is not null.
Attachments
Issue Links
- is caused by
-
TEZ-4135 Improve memory allocation when executing in-memory reads
- Resolved
- links to