Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-26444

Use tez.local.mode.without.network for unit tests

    XMLWordPrintableJSON

Details

    Description

      Since TEZ-4236 (in Tez 0.10.1), tez local mode can run without even starting an RPC server in the DAGAppMaster, which is in the same JVM as the client.
      Adapting tez.local.mode.without.network=true could make tez.local.mode=true unit tests more stable.

      here is an example where I had no idea why the dag app master connection was refused:

      2022-07-29T07:56:24,701  INFO [main_executor] ql.Driver: Executing command(queryId=jenkins_20220729075624_b3ba4c8a-82d5-4ebd-b4b0-218325a71b10): INSERT into table default.tmp_minor_compactor_testmmminorcompaction_1659106584519_result select `a`, `b` from default.tmp_minor_compactor_testmmminorcompaction_1659106584519
      
      2022-07-29T07:56:24,823  INFO [ServiceThread:DAGClientRPCServer] client.DAGClientServer: Instantiated DAGClientRPCServer at internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19:22623
      2022-07-29T07:56:24,823  INFO [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] rm.TaskSchedulerManager: Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1659106584728
      2022-07-29T07:56:24,825  INFO [DAGAppMaster Thread] HistoryEventHandler.criticalEvents: [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1659106584728_0001_000000, startTime=1659106584825
      2022-07-29T07:56:24,825  INFO [DAGAppMaster Thread] app.DAGAppMaster: In Session mode. Waiting for DAG over RPC
      2022-07-29T07:56:24,871  INFO [main_executor] client.LocalClient: DAGAppMaster state: IDLE
      2022-07-29T07:56:24,871  INFO [main_executor] client.TezClient: The url to track the Tez Session: N/A
      ...
      2022-07-29T07:56:46,384  INFO [main_executor] client.TezClient: Failed to retrieve AM Status via proxy
      com.google.protobuf.ServiceException: java.net.ConnectException: Call From internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19 to internal-hive-flaky-check-88-xwmrs-v2h77-knnxx:22623 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:247) ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
      	at com.sun.proxy.$Proxy50.getAMStatus(Unknown Source) ~[?:?]
      

      instead of diving deep into an evil environment related bug, we can simply utilize TEZ-4236 in these cases too

      Attachments

        Issue Links

          Activity

            People

              abstractdog László Bodor
              abstractdog László Bodor
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h