Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7926 long-lived daemons for query fragment execution, I/O and caching
  3. HIVE-11524

LLAP: tez.runtime.compress doesn't appear to be honored for LLAP

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Cannot Reproduce
    • None
    • llap
    • llap
    • None

    Description

      When running llap on an openstack cluster without snappy installed, with tez.runtime.compress set to false and codec set to snappy, one still gets the exceptions due to snappy codec being absent:

      2015-08-10 11:14:30,440 [TezTaskRunner_attempt_1438943112941_0015_2_00_000000_0(attempt_1438943112941_0015_2_00_000000_0)] ERROR org.apache.hadoop.io.compress.snappy.SnappyCompressor: failed to load SnappyCompressor
      java.lang.NoSuchFieldError: clazz
      	at org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs(Native Method)
      	at org.apache.hadoop.io.compress.snappy.SnappyCompressor.<clinit>(SnappyCompressor.java:57)
      	at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:69)
      	at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
      	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
      	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
      	at org.apache.tez.runtime.library.common.sort.impl.IFile$Writer.<init>(IFile.java:153)
      	at org.apache.tez.runtime.library.common.sort.impl.IFile$Writer.<init>(IFile.java:138)
      	at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter$SpillCallable.callInternal(UnorderedPartitionedKVWriter.java:406)
      	at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter$SpillCallable.callInternal(UnorderedPartitionedKVWriter.java:367)
      	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      	at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.finalSpill(UnorderedPartitionedKVWriter.java:612)
      	at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.close(UnorderedPartitionedKVWriter.java:521)
      	at org.apache.tez.runtime.library.output.UnorderedKVOutput.close(UnorderedKVOutput.java:128)
      	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.close(LogicalIOProcessorRuntimeTask.java:376)
      	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:79)
      	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1655)
      	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
      	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
      	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      When it's set to true, the client complains about snappy. When it's set to fails, the client doesn't complain but it still tries to use it.

      Attachments

        Activity

          People

            sseth Siddharth Seth
            sershe Sergey Shelukhin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: