Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-25565

Materialized view Rebuild of Non-Txn table issue Aws EMR

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.2
    • None
    • Materialized views
    • None
    • Aws EMR - 6.3.0

      Spark -  3.1.1

      Hive - 3.1.2

    Description

      We have Materialized views built on top of Hudi tables which are hive-sync'd.
      Hive uses AWS Glue for its metastore catalog. 

      We are running into issue whenever we are trying to "*rebuild*" Hive materialized views.

      Please note, creation of materialized views works fine.  It's only rebuild which is failing.

      However, it does seem the rebuild actually seems to work behind the scenes but its throws some exception causing EMR steps to fail. 
      Can anyone please guide us here, about any config changes that we need to do or anything. Any help will be great.  

       

      The stack trace of the exception - 

       FAILED: Hive Internal Error: org.apache.hadoop.hive.ql.metadata.HiveException(Error while invoking FailureHook. hooks: java.lang.NullPointerException at org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45) at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296) at org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283) at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224) at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)) > org.apache.hadoop.hive.ql.metadata.HiveException: Error while invoking FailureHook. hooks:  > java.lang.NullPointerException > at org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)> at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)> at org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)> at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616)> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386)> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)> at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)> at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)> at java.security.AccessController.doPrivileged(Native Method)> at javax.security.auth.Subject.doAs(Subject.java:422)> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)> at java.util.concurrent.FutureTask.run(FutureTask.java:266)> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)> at java.lang.Thread.run(Thread.java:748)> > at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:302)> at org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)> at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616)> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386)> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)> at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)> at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)> at java.security.AccessController.doPrivileged(Native Method)> at javax.security.auth.Subject.doAs(Subject.java:422)> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)> at java.util.concurrent.FutureTask.run(FutureTask.java:266)> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)> at java.lang.Thread.run(Thread.java:748)> Caused by: java.lang.NullPointerException> at org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)> at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)> ... 19 more > INFO  : Completed executing command(queryId=hive_20210927174227_c0ccf861-e7df-4ae4-be7d-b93774d29c25); Time taken: 15.402 seconds> INFO  : Concurrency mode is disabled, not creating a lock manager> Error: Error while processing statement: FAILED: Hive Internal Error: org.apache.hadoop.hive.ql.metadata.HiveException(Error while invoking FailureHook. hooks: java.lang.NullPointerException> at org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)> at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)> at org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)> at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616)> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386)> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)> at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)> at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)> at java.security.AccessController.doPrivileged(Native Method)> at javax.security.auth.Subject.doAs(Subject.java:422)> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)> at java.util.concurrent.FutureTask.run(FutureTask.java:266)> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)> at java.lang.Thread.run(Thread.java:748)> ) (state=08S01,code=12)> 

      Attachments

        Activity

          People

            Unassigned Unassigned
            vipina Vipin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: