Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-5267

With namespaces enabled Phoenix client times out with high loads

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 4.14.1
    • 4.14.2
    • None
    • None

    Description

      Steps to reproduce:

      • Enable namespaces for phoenix 4.14.1 and hbase 1.3
      • Run high load using pherf client with 48 threads

      After sometime the client hangs. and gives timeout exception

      [pool-1-thread-1] WARN org.apache.phoenix.pherf.workload.WriteWorkload -

java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: callTimeout=1200000, callDuration=1238263: Call to <host> failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=857, waitTime=120001, operationTimeout=120000 expired. row '^@TEST^@TABLE' on table 'SYSTEM:CATALOG' at region=SYSTEM:CATALOG,1556024429507.0f80d6de0a002d1421b8fd384e956254., hostname=<host>, seqNum=2

    at java.util.concurrent.FutureTask.report(FutureTask.java:122)

    at java.util.concurrent.FutureTask.get(FutureTask.java:192)

    at org.apache.phoenix.pherf.workload.WriteWorkload.waitForBatches(WriteWorkload.java:239)

    at org.apache.phoenix.pherf.workload.WriteWorkload.exec(WriteWorkload.java:189)

    at org.apache.phoenix.pherf.workload.WriteWorkload.access$100(WriteWorkload.java:56)

    at org.apache.phoenix.pherf.workload.WriteWorkload$1.run(WriteWorkload.java:165)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

    at java.util.concurrent.FutureTask.run(FutureTask.java:266)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

    at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.phoenix.exception.PhoenixIOException: callTimeout=1200000, callDuration=1238263: Call to <host> failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=857, waitTime=120001, operationTimeout=120000 expired. row '^@TEST^@TABLE' on table 'SYSTEM:CATALOG' at region=SYSTEM:CATALOG,1556024429507.0f80d6de0a002d1421b8fd384e956254., hostname=<host>, seqNum=2

    at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)

    at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1379)

    at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1343)

    at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1560)

    at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:644)

    at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:538)

    at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:530)

    at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:526)

    at org.apache.phoenix.execute.MutationState.validateAndGetServerTimestamp(MutationState.java:755)

    at org.apache.phoenix.execute.MutationState.validateAll(MutationState.java:743)

    at org.apache.phoenix.execute.MutationState.send(MutationState.java:875)

    at org.apache.phoenix.execute.MutationState.send(MutationState.java:1360)

    at org.apache.phoenix.execute.MutationState.commit(MutationState.java:1183)

    at org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:670)

    at org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:666)

    at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)

    at org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConnection.java:666)

    at org.apache.phoenix.pherf.workload.WriteWorkload$2.call(WriteWorkload.java:297)

    at org.apache.phoenix.pherf.workload.WriteWorkload$2.call(WriteWorkload.java:256)

      Attachments

        Issue Links

          is superceded by

          Bug - A problem which impairs or prevents the functions of the product. PHOENIX-5269 PhoenixAccessController should use AccessChecker instead of AccessControlClient for permission checks

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. PHOENIX-5269 PhoenixAccessController should use AccessChecker instead of AccessControlClient for permission checks

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              Unassigned Unassigned
              kiran.maturi Kiran Kumar Maturi
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: