Details
-
Bug
-
Status: Open
-
P3
-
Resolution: Unresolved
-
2.8.0, 2.9.0
-
None
-
None
-
CentOS Linux release 7.6.1810 (also tried Ubuntu 18.10)
flink-1.5.3 (standalone, without changing any conf, just download and untar, then kick off by ./start-cluster.sh)
Apache Beam, release-2.9.0 branch (also tried 2.8.0 branch)
Docker version 18.09.0 (also tried the old 1.13.1)
java version "1.8.0_111"
Python 2.7.5CentOS Linux release 7.6.1810 (also tried Ubuntu 18.10) flink-1.5.3 (standalone, without changing any conf, just download and untar, then kick off by ./start-cluster.sh) Apache Beam, release-2.9.0 branch (also tried 2.8.0 branch) Docker version 18.09.0 (also tried the old 1.13.1) java version "1.8.0_111" Python 2.7.5
Description
Recently I tried to run portable python wordcount example following exactly what it says in the guide page https://beam.apache.org/roadmap/portability/, but still no luck after several days trying.
Here're my steps,
1. ./gradlew :beam-sdks-python-container:docker (to build the sdk container, no error)
2. ./gradlew :beam-runners-flink_2.11-job-server:installShadowDist (to build jobservice jar, no error)
3. ./gradlew :beam-runners-flink_2.11-job-server:runShadow -PflinkMasterUrl=localhost:8081 (to run jobservice, no error)
4. ./gradlew :beam-sdks-python:portableWordCount -PjobEndpoint=localhost:8099 (to submit wordcount job, error as mentioned below)
Have seen 2 errors, sometime, it says,
BeamIOError: src and dst files do not exist. src: /tmp/beam-temp-py-wordcount-direct-56d037a2ffa611e88f9208002721555c/020e033b-ec4f-495f-bfa4-e49b59ef9d36.py-wordcount-direct, dst: /tmp/py-wordcount-direct-00000-of-00001 with exceptions None [while running 'write/Write/WriteImpl/FinalizeWrite'] with exceptions None
Or it says,
RuntimeError: IOError: [Errno 2] No such file or directory: '/tmp/beam-temp-py-wordcount-direct-7013276afc6311e8a71108002721555c/72786d7f-a2b4-461c-91fa-c7e04d55ea3a.py-wordcount-direct' [while running 'write/Write/WriteImpl/WriteBundles']