Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7054

PortableRunner on Flink cluster crashes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.11.0
    • Fix Version/s: 2.11.0
    • Labels:
      None
    • Environment:
      Ubuntu 18.04, Beam 2.11.0, Flink 1.7.2, Docker 18.09.4, on a 2-node cluster with identical VM images

      Description

      Setup

      2-node cluster: `master` and `worker`. `master` runs Flink JobManager, `worker` runs Flink TaskManager.

      Problem description

      When running Flink TaskManager on `master` and running the word count example provided by Beam as

      python sdks/python/apache_beam/examples/wordcount.py --input=gs://<input file location>-output=gs://<output file location>--runner=PortableRunner --job_endpoint=localhost:8099 --environment_config='<docker image location>'
      

      on `master`, the job executes correctly.

      However, when running the Flink TaskManger on `worker` (the second node), the `worker` outputs:

      java.util.concurrent.ExecutionException: java.io.FileNotFoundException: /tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST (No such file or directory)
      

      and crashes (full log in [1]).

      The docker image has been created via:

      ./gradlew :beam-sdks-python-container:docker
      

      The job server has been set up on `master` via:

      ./gradlew :beam-runners-flink-1.7-job-server:runShadow -PflinkMasterUrl=localhost:8081
      

      which logs

      > Task :beam-runners-flink-1.7-job-server:runShadow
      
      Listening for transport dt_socket at address: 5005
      
      [main] INFO org.apache.beam.runners.flink.FlinkJobServerDriver - ArtifactStagingService started on localhost:8098
      
      [main] INFO org.apache.beam.runners.flink.FlinkJobServerDriver - JobService started on localhost:8099
      
      <============-> 98% EXECUTING [3m 0s]
      
      > :beam-runners-flink-1.7-job-server:runShadow
      
      > IDLE
      
      > IDLE
      
      > IDLE
      

      The default Flink example

      ./bin/flink run ./examples/batch/WordCount.jarworks correctly on the 2-node cluster.
      

      works correctly on the 2-node cluster. 

      Expected behavior

      Either the job executes correctly and returns as in the single-node setup or additional configuration is needed, which is not reflected in the docs. Please let me know if I missed something.

       


       

      [1] 

      tail -111111f /root/flink-1.7.2/log/flink-root-taskexecutor-0-VM-0-16-ubuntu.log 
      
      2019-04-11 13:11:59,497 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - --------------------------------------------------------------------------------
      
      2019-04-11 13:11:59,498 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  Starting TaskManager (Version: 1.7.2, Rev:ceba8af, Date:11.02.2019 @ 14:17:09 UTC)
      
      2019-04-11 13:11:59,498 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  OS current user: root
      
      2019-04-11 13:11:59,499 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  Current Hadoop/Kerberos user: <no hadoop dependency found>
      
      2019-04-11 13:11:59,499 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.191-b12
      
      2019-04-11 13:11:59,499 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  Maximum heap size: 922 MiBytes
      
      2019-04-11 13:11:59,499 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  JAVA_HOME: (not set)
      
      2019-04-11 13:11:59,499 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  No Hadoop Dependency available
      
      2019-04-11 13:11:59,499 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  JVM Options:
      
      2019-04-11 13:11:59,499 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -XX:+UseG1GC
      
      2019-04-11 13:11:59,499 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -Xms922M
      
      2019-04-11 13:11:59,499 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -Xmx922M
      
      2019-04-11 13:11:59,500 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -XX:MaxDirectMemorySize=8388607T
      
      2019-04-11 13:11:59,500 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -Dlog.file=/root/flink-1.7.2/log/flink-root-taskexecutor-0-VM-0-16-ubuntu.log
      
      2019-04-11 13:11:59,500 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -Dlog4j.configuration=file:/root/flink-1.7.2/conf/log4j.properties
      
      2019-04-11 13:11:59,500 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     -Dlogback.configurationFile=file:/root/flink-1.7.2/conf/logback.xml
      
      2019-04-11 13:11:59,500 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  Program Arguments:
      
      2019-04-11 13:11:59,500 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     --configDir
      
      2019-04-11 13:11:59,500 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -     /root/flink-1.7.2/conf
      
      2019-04-11 13:11:59,500 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       -  Classpath: /root/flink-1.7.2/lib/flink-python_2.11-1.7.2.jar:/root/flink-1.7.2/lib/log4j-1.2.17.jar:/root/flink-1.7.2/lib/slf4j-log4j12-1.7.15.jar:/root/flink-1.7.2/lib/flink-dist_2.11-1.7.2.jar:::
      
      2019-04-11 13:11:59,500 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - --------------------------------------------------------------------------------
      
      2019-04-11 13:11:59,502 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - Registered UNIX signal handlers for [TERM, HUP, INT]
      
      2019-04-11 13:11:59,505 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - Maximum number of open file descriptors is 1048576.
      
      2019-04-11 13:11:59,517 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, 172.26.0.4
      
      2019-04-11 13:11:59,517 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
      
      2019-04-11 13:11:59,517 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 1024m
      
      2019-04-11 13:11:59,517 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 1024m
      
      2019-04-11 13:11:59,517 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 4
      
      2019-04-11 13:11:59,517 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 2
      
      2019-04-11 13:11:59,518 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 8081
      
      2019-04-11 13:11:59,523 INFO  org.apache.flink.core.fs.FileSystem                           - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
      
      2019-04-11 13:11:59,547 INFO  org.apache.flink.runtime.security.modules.HadoopModuleFactory  - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
      
      2019-04-11 13:11:59,568 INFO  org.apache.flink.runtime.security.SecurityUtils               - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
      
      2019-04-11 13:11:59,754 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'
      
      2019-04-11 13:11:59,758 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils            - Trying to select the network interface and address to use by connecting to the leading JobManager.
      
      2019-04-11 13:11:59,759 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils            - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics
      
      2019-04-11 13:11:59,762 INFO  org.apache.flink.runtime.net.ConnectionUtils                  - Retrieved new target address /172.26.0.4:6123.
      
      2019-04-11 13:11:59,769 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - TaskManager will use hostname/address '172.26.0.16' (172.26.0.16) for communication.
      
      2019-04-11 13:11:59,772 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Trying to start actor system at 172.26.0.16:0
      
      2019-04-11 13:12:00,135 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
      
      2019-04-11 13:12:00,201 INFO  akka.remote.Remoting                                          - Starting remoting
      
      2019-04-11 13:12:00,303 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink@172.26.0.16:39493]
      
      2019-04-11 13:12:00,310 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Actor system started at akka.tcp://flink@172.26.0.16:39493
      
      2019-04-11 13:12:00,316 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - Trying to start actor system at 172.26.0.16:0
      
      2019-04-11 13:12:00,332 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
      
      2019-04-11 13:12:00,338 INFO  akka.remote.Remoting                                          - Starting remoting
      
      2019-04-11 13:12:00,354 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - Actor system started at akka.tcp://flink-metrics@172.26.0.16:41573
      
      2019-04-11 13:12:00,356 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink-metrics@172.26.0.16:41573]
      
      2019-04-11 13:12:00,367 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - No metrics reporter configured, no metrics will be exposed/reported.
      
      2019-04-11 13:12:00,373 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Created BLOB cache storage directory /tmp/blobStore-b1adf8f9-213f-4fcf-86ee-be140bc76181
      
      2019-04-11 13:12:00,376 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Created BLOB cache storage directory /tmp/blobStore-3bdadd52-faad-4807-b06a-6e6924f1b1d0
      
      2019-04-11 13:12:00,380 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - Starting TaskManager with ResourceID: 76eff45c35decbc12d43e53d88883a7f
      
      2019-04-11 13:12:00,384 INFO  org.apache.flink.runtime.io.network.netty.NettyConfig         - NettyConfig [server address: /172.26.0.16, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 4 (manual), number of client threads: 4 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]
      
      2019-04-11 13:12:00,423 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerServices     - Temporary file directory '/tmp': total 49 GB, usable 38 GB (77.55% usable)
      
      2019-04-11 13:12:00,492 INFO  org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated 102 MB for network buffer pool (number of memory segments: 3278, bytes per segment: 32768).
      
      2019-04-11 13:12:00,542 INFO  org.apache.flink.runtime.query.QueryableStateUtils            - Could not load Queryable State Client Proxy. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder.
      
      2019-04-11 13:12:00,542 INFO  org.apache.flink.runtime.query.QueryableStateUtils            - Could not load Queryable State Server. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder.
      
      2019-04-11 13:12:00,543 INFO  org.apache.flink.runtime.io.network.NetworkEnvironment        - Starting the network environment and its components.
      
      2019-04-11 13:12:00,578 INFO  org.apache.flink.runtime.io.network.netty.NettyClient         - Successful initialization (took 33 ms).
      
      2019-04-11 13:12:00,624 INFO  org.apache.flink.runtime.io.network.netty.NettyServer         - Successful initialization (took 45 ms). Listening on SocketAddress /172.26.0.16:33565.
      
      2019-04-11 13:12:00,625 WARN  org.apache.flink.runtime.taskmanager.TaskManagerLocation      - No hostname could be resolved for the IP address 172.26.0.16, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted.
      
      2019-04-11 13:12:00,625 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerServices     - Limiting managed memory to 0.7 of the currently free heap space (640 MB), memory will be allocated lazily.
      
      2019-04-11 13:12:00,629 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager uses directory /tmp/flink-io-eb728574-61d9-4170-a020-5eeed20b8551 for spill files.
      
      2019-04-11 13:12:00,693 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration  - Messages have a max timeout of 10000 ms
      
      2019-04-11 13:12:00,701 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/taskmanager_0 .
      
      2019-04-11 13:12:00,717 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService        - Start job leader service.
      
      2019-04-11 13:12:00,718 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Connecting to ResourceManager akka.tcp://flink@172.26.0.4:6123/user/resourcemanager(00000000000000000000000000000000).
      
      2019-04-11 13:12:00,718 INFO  org.apache.flink.runtime.filecache.FileCache                  - User file cache uses directory /tmp/flink-dist-cache-d80719fe-a81a-47b6-9560-59766fb86afe
      
      2019-04-11 13:12:00,907 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Resolved ResourceManager address, beginning registration
      
      2019-04-11 13:12:00,907 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Registration at ResourceManager attempt 1 (timeout=100ms)
      
      2019-04-11 13:12:00,965 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Successful registration at resource manager akka.tcp://flink@172.26.0.4:6123/user/resourcemanager under registration id 28630de4ae4181854d3b7143d9a37af1.
      
      2019-04-11 13:22:13,394 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Receive slot request AllocationID\{8810509925a2184bdabc6d58bd808b17} for job a0f3ed507058c39f6261f1e8953d9363 from resource manager with leader id 00000000000000000000000000000000.
      
      2019-04-11 13:22:13,395 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Allocated slot for AllocationID\{8810509925a2184bdabc6d58bd808b17}.
      
      2019-04-11 13:22:13,396 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService        - Add job a0f3ed507058c39f6261f1e8953d9363 for job leader monitoring.
      
      2019-04-11 13:22:13,397 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService        - Try to register at job manager akka.tcp://flink@172.26.0.4:6123/user/jobmanager_0 with leader id 00000000-0000-0000-0000-000000000000.
      
      2019-04-11 13:22:13,414 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService        - Resolved JobManager address, beginning registration
      
      2019-04-11 13:22:13,414 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService        - Registration at JobManager attempt 1 (timeout=100ms)
      
      2019-04-11 13:22:13,433 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService        - Successful registration at job manager akka.tcp://flink@172.26.0.4:6123/user/jobmanager_0 for job a0f3ed507058c39f6261f1e8953d9363.
      
      2019-04-11 13:22:13,434 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Establish JobManager connection for job a0f3ed507058c39f6261f1e8953d9363.
      
      2019-04-11 13:22:13,437 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Offer reserved slots to the leader of job a0f3ed507058c39f6261f1e8953d9363.
      
      2019-04-11 13:22:13,454 INFO  org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable      - Activate slot AllocationID\{8810509925a2184bdabc6d58bd808b17}.
      
      2019-04-11 13:22:13,481 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Received task DataSource (Impulse) (1/1).
      
      2019-04-11 13:22:13,482 INFO  org.apache.flink.runtime.taskmanager.Task                     - DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) switched from CREATED to DEPLOYING.
      
      2019-04-11 13:22:13,482 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Received task DataSource (Impulse) (1/1).
      
      2019-04-11 13:22:13,483 INFO  org.apache.flink.runtime.taskmanager.Task                     - Creating FileSystem stream leak safety net for task DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) [DEPLOYING]
      
      2019-04-11 13:22:13,483 INFO  org.apache.flink.runtime.taskmanager.Task                     - DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) switched from CREATED to DEPLOYING.
      
      2019-04-11 13:22:13,484 INFO  org.apache.flink.runtime.taskmanager.Task                     - Creating FileSystem stream leak safety net for task DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) [DEPLOYING]
      
      2019-04-11 13:22:13,486 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) [DEPLOYING].
      
      2019-04-11 13:22:13,487 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) [DEPLOYING].
      
      2019-04-11 13:22:13,489 INFO  org.apache.flink.runtime.blob.BlobClient                      - Downloading a0f3ed507058c39f6261f1e8953d9363/p-f784164b3f975e48e5bdfa7f321e70fbe4b0e186-ca919340fcac9cb4b2e93b7b6225711b from /172.26.0.4:45541
      
      2019-04-11 13:22:14,085 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) [DEPLOYING].
      
      2019-04-11 13:22:14,085 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) [DEPLOYING].
      
      2019-04-11 13:22:14,103 INFO  org.apache.flink.runtime.taskmanager.Task                     - DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) switched from DEPLOYING to RUNNING.
      
      2019-04-11 13:22:14,103 INFO  org.apache.flink.runtime.taskmanager.Task                     - DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) switched from DEPLOYING to RUNNING.
      
      2019-04-11 13:22:14,325 INFO  org.apache.flink.runtime.taskmanager.Task                     - DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) switched from RUNNING to FINISHED.
      
      2019-04-11 13:22:14,325 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000).
      
      2019-04-11 13:22:14,326 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) [FINISHED]
      
      2019-04-11 13:22:14,326 INFO  org.apache.flink.runtime.taskmanager.Task                     - DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) switched from RUNNING to FINISHED.
      
      2019-04-11 13:22:14,326 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1).
      
      2019-04-11 13:22:14,327 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) [FINISHED]
      
      2019-04-11 13:22:14,327 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state FINISHED to JobManager for task DataSource (Impulse) 94014197941e34830c0d79aff847f000.
      
      2019-04-11 13:22:14,333 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state FINISHED to JobManager for task DataSource (Impulse) ad6774374f4abba501b8f319af3d9ac1.
      
      2019-04-11 13:22:14,341 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Received task CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1).
      
      2019-04-11 13:22:14,343 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Received task CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1).
      
      2019-04-11 13:22:14,343 INFO  org.apache.flink.runtime.taskmanager.Task                     - CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c3350586427cb04999d1c23038e2b1fc) switched from CREATED to DEPLOYING.
      
      2019-04-11 13:22:14,344 INFO  org.apache.flink.runtime.taskmanager.Task                     - Creating FileSystem stream leak safety net for task CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c3350586427cb04999d1c23038e2b1fc) [DEPLOYING]
      
      2019-04-11 13:22:14,348 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c3350586427cb04999d1c23038e2b1fc) [DEPLOYING].
      
      2019-04-11 13:22:14,350 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c3350586427cb04999d1c23038e2b1fc) [DEPLOYING].
      
      2019-04-11 13:22:14,350 INFO  org.apache.flink.runtime.taskmanager.Task                     - CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c3350586427cb04999d1c23038e2b1fc) switched from DEPLOYING to RUNNING.
      
      2019-04-11 13:22:14,352 WARN  org.apache.flink.metrics.MetricGroup                          - The operator name MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) exceeded the 80 characters length limit and was truncated.
      
      2019-04-11 13:22:14,352 INFO  org.apache.flink.runtime.taskmanager.Task                     - CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c9bb2f2368faf10656e3ef9ce40e578f) switched from CREATED to DEPLOYING.
      
      2019-04-11 13:22:14,353 INFO  org.apache.flink.runtime.taskmanager.Task                     - Creating FileSystem stream leak safety net for task CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c9bb2f2368faf10656e3ef9ce40e578f) [DEPLOYING]
      
      2019-04-11 13:22:14,356 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c9bb2f2368faf10656e3ef9ce40e578f) [DEPLOYING].
      
      2019-04-11 13:22:14,356 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c9bb2f2368faf10656e3ef9ce40e578f) [DEPLOYING].
      
      2019-04-11 13:22:14,365 INFO  org.apache.flink.runtime.taskmanager.Task                     - CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c9bb2f2368faf10656e3ef9ce40e578f) switched from DEPLOYING to RUNNING.
      
      2019-04-11 13:22:15,156 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, 172.26.0.4
      
      2019-04-11 13:22:15,156 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
      
      2019-04-11 13:22:15,156 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 1024m
      
      2019-04-11 13:22:15,156 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 1024m
      
      2019-04-11 13:22:15,156 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 4
      
      2019-04-11 13:22:15,156 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 2
      
      2019-04-11 13:22:15,157 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 8081
      
      2019-04-11 13:22:16,544 INFO  org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService  - GetManifest for /tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST
      
      2019-04-11 13:22:16,545 INFO  org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService  - Loading manifest for retrieval token /tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST
      
      2019-04-11 13:22:16,552 INFO  org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService  - GetManifest for /tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST failed
      
      java.util.concurrent.ExecutionException: java.io.FileNotFoundException: /tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST (No such file or directory)
      
      at org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
      
      at org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:459)
      
      at org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:76)
      
      at org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:142)
      
      at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2373)
      
      at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2337)
      
      at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
      
      at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
      
      at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
      
      at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
      
      at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
      
      at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.getManifest(BeamFileSystemArtifactRetrievalService.java:80)
      
      at org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc$MethodHandlers.invoke(ArtifactRetrievalServiceGrpc.java:298)
      
      at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
      
      at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
      
      at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
      
      at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
      
      at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)
      
      at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
      
      at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
      
      at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
      
      at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
      
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      
      at java.lang.Thread.run(Thread.java:748)
      
      Caused by: java.io.FileNotFoundException: /tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST (No such file or directory)
      
      at java.io.FileInputStream.open0(Native Method)
      
      at java.io.FileInputStream.open(FileInputStream.java:195)
      
      at java.io.FileInputStream.<init>(FileInputStream.java:138)
      
      at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:114)
      
      at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:81)
      
      at org.apache.beam.sdk.io.FileSystems.open(FileSystems.java:251)
      
      at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.loadManifest(BeamFileSystemArtifactRetrievalService.java:185)
      
      at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.loadManifest(BeamFileSystemArtifactRetrievalService.java:180)
      
      at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService$1.load(BeamFileSystemArtifactRetrievalService.java:171)
      
      at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService$1.load(BeamFileSystemArtifactRetrievalService.java:168)
      
      at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
      
      at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
      
      ... 19 more
      

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rein Rein Houthooft
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: