Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7993

portable python precommit is flaky

Details

    Description

      I'm not sure what the root cause is here.

      Example log where :sdks:python:test-suites:portable:py35:portableWordCountBatch failed:

      11:51:22 [CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (2/2)] ERROR org.apache.flink.runtime.operators.BatchTask - Error in task code:  CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (2/2)
      11:51:22 [CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/2)] ERROR org.apache.flink.runtime.operators.BatchTask - Error in task code:  CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/2)
      11:51:22 [CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/{FlatMap(<lambda at core.py:2457>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (2/2)] ERROR org.apache.flink.runtime.operators.BatchTask - Error in task code:  CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/{FlatMap(<lambda at core.py:2457>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (2/2)
      11:51:22 [CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/{FlatMap(<lambda at core.py:2457>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/2)] ERROR org.apache.flink.runtime.operators.BatchTask - Error in task code:  CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/{FlatMap(<lambda at core.py:2457>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/2)
      11:51:22 java.lang.Exception: The user defined 'open()' method caused an exception: java.io.IOException: Received exit code 1 for command 'docker inspect -f {{.State.Running}} 642c312c335d3881b885873c66917b536e79cff07503fdceaddee5fbeb10bfd1'. stderr: Error: No such object: 642c312c335d3881b885873c66917b536e79cff07503fdceaddee5fbeb10bfd1
      11:51:22 	at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:498)
      11:51:22 	at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
      11:51:22 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
      11:51:22 	at java.lang.Thread.run(Thread.java:748)
      11:51:22 Caused by: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.io.IOException: Received exit code 1 for command 'docker inspect -f {{.State.Running}} 642c312c335d3881b885873c66917b536e79cff07503fdceaddee5fbeb10bfd1'. stderr: Error: No such object: 642c312c335d3881b885873c66917b536e79cff07503fdceaddee5fbeb10bfd1
      11:51:22 	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4966)
      11:51:22 	at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.<init>(DefaultJobBundleFactory.java:211)
      11:51:22 	at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.<init>(DefaultJobBundleFactory.java:202)
      11:51:22 	at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory.forStage(DefaultJobBundleFactory.java:185)
      11:51:22 	at org.apache.beam.runners.flink.translation.functions.FlinkDefaultExecutableStageContext.getStageBundleFactory(FlinkDefaultExecutableStageContext.java:49)
      11:51:22 	at org.apache.beam.runners.flink.translation.functions.ReferenceCountingFlinkExecutableStageContextFactory$WrappedContext.getStageBundleFactory(ReferenceCountingFlinkExecutableStageContextFactory.java:203)
      11:51:22 	at org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction.open(FlinkExecutableStageFunction.java:129)
      11:51:22 	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
      11:51:22 	at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:494)
      11:51:22 	... 3 more
      

      https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/5512/consoleFull

      Attachments

        1. Python_Portable_Precommit.pdf
          49 kB
          Hannah Jiang

        Issue Links

          Activity

            People

              markflyhigh Mark Liu
              udim Udi Meiri
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 5.5h
                  5.5h