Details
-
Bug
-
Status: Open
-
P3
-
Resolution: Unresolved
-
2.27.0
-
None
-
Google Cloud.
A list of linked to the dataflow libraries:
proto-google-cloud-spanner-v1-2.0.2.jar
grpc-google-cloud-bigquerystorage-v1beta1-0.105.5.jar
commons-io-2.6.jar
grpc-alts-1.34.0.jar
beam-runners-core-construction-java-2.27.0.jar
beam-vendor-grpc-1_26_0-0.3.jar
proto-google-cloud-spanner-admin-instance-v1-2.0.2.jar
vavr-0.10.2.jar
jackson-dataformat-yaml-2.12.0.jar
grpc-api-1.34.0.jar
jaxb-api-2.3.0.jar
hamcrest-core-2.1.jar
grpc-netty-1.32.2.jar
kamon-bundle_2.12-2.1.4.jar
beam-sdks-java-extensions-protobuf-2.27.0.jar
grpc-netty-shaded-1.34.0.jar
jackson-databind-2.12.0.jar
api-common-1.10.1.jar
paranamer-2.7.jar
google-cloud-bigquery-1.122.2.jar
google-auth-library-credentials-0.22.0.jar
log4j-api-2.6.2.jar
google-api-services-cloudresourcemanager-v1-rev20200720-1.30.10.jar
proto-google-cloud-spanner-admin-database-v1-2.0.2.jar
netty-codec-http2-4.1.51.Final.jar
perfmark-api-0.19.0.jar
netty-handler-4.1.51.Final.jar
checker-compat-qual-2.5.5.jar
beam-vendor-sdks-java-extensions-protobuf-2.27.0.jar
beam-model-fn-execution-2.27.0.jar
junit-4.13.1.jar
google-api-services-bigquery-v2-rev20200916-1.30.10.jar
netty-tcnative-boringssl-static-2.0.33.Final.jar
resilience4j-core-1.6.1.jar
kamon-prometheus_2.12-2.1.4.jar
netty-common-4.1.51.Final.jar
google-cloud-core-1.93.9.jar
google-http-client-jackson2-1.38.0.jar
grpc-google-cloud-spanner-v1-2.0.2.jar
google-api-client-java6-1.30.10.jar
proto-google-iam-v1-1.0.3.jar
grpc-stub-1.34.0.jar
classgraph-4.8.65.jar
beam-sdks-java-fn-execution-2.27.0.jar
commons-lang3-3.5.jar
gax-1.60.1.jar
proto-google-cloud-bigquerystorage-v1beta1-0.105.5.jar
jsr305-3.0.2.jar
grpc-google-cloud-bigtable-admin-v2-1.14.0.jar
proto-google-cloud-bigquerystorage-v1alpha2-0.105.5.jar
opencensus-api-0.24.0.jar
beam-sdks-java-extensions-google-cloud-platform-core-2.27.0.jar
opencensus-contrib-grpc-util-0.24.0.jar
protobuf-java-util-3.14.0.jar
netty-buffer-4.1.51.Final.jar
google-api-services-pubsub-v1-rev20200713-1.30.10.jar
httpclient-cache-4.5.9.jar
google-api-services-storage-v1-rev20200927-1.30.10.jar
google-auth-library-oauth2-http-0.22.0.jar
proto-google-cloud-bigquerystorage-v1-1.5.5.jar
simpleclient-0.9.0.jar
auto-value-annotations-1.7.4.jar
grpc-google-cloud-pubsublite-v1-0.7.0.jar
metrics-core-3.2.6.jar
listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
beam-sdks-java-core-2.27.0.jar
google-cloud-bigquerystorage-1.5.5.jar
error_prone_annotations-2.4.0.jar
xz-1.8.jar
protobuf-java-3.14.0.jar
slf4j-api-1.7.30.jar
grpc-auth-1.34.0.jar
jna-platform-5.5.0.jar
google-api-services-clouddebugger-v2-rev20200501-1.30.10.jar
scala-library-2.12.11.jar
datastore-v1-proto-client-1.6.3.jar
joda-time-2.10.5.jar
grpc-google-cloud-bigquerystorage-v1beta2-0.105.5.jar
okhttp-3.14.7.jar
google-api-services-healthcare-v1beta1-rev20200713-1.30.10.jar
proto-google-common-protos-2.0.1.jar
proto-google-cloud-bigtable-admin-v2-1.14.0.jar
jackson-annotations-2.12.0.jar
conscrypt-openjdk-uber-2.5.1.jar
config-1.4.0.jar
jackson-mapper-asl-1.9.13.jar
commons-codec-1.14.jar
grpc-context-1.34.0.jar
args4j-2.33.jar
commons-compress-1.20.jar
json-20200518.jar
grpc-google-cloud-spanner-admin-instance-v1-2.0.2.jar
animal-sniffer-annotations-1.19.jar
netty-handler-proxy-4.1.51.Final.jar
logback-classic-1.2.3.jar
google-api-client-jackson2-1.30.10.jar
logstash-logback-encoder-6.2.jar
logback-core-1.2.3.jar
bigtable-client-core-1.16.0.jar
threetenbp-1.5.0.jar
snakeyaml-1.27.jar
opencensus-contrib-http-util-0.24.0.jar
google-http-client-1.38.0.jar
google-cloud-pubsub-1.110.0.jar
resilience4j-retry-1.6.1.jar
gcsio-2.1.6.jar
beam-vendor-bytebuddy-1_10_8-0.1.jar
google-cloud-bigtable-1.14.0.jar
proto-google-cloud-datastore-v1-0.88.0.jar
grpc-google-common-protos-1.18.1.jar
jackson-core-asl-1.9.13.jar
util-2.1.6.jar
checker-qual-3.7.0.jar
google-extensions-0.5.1.jar
httpclient-4.5.13.jar
netty-resolver-4.1.51.Final.jar
netty-transport-4.1.51.Final.jar
grpc-google-cloud-spanner-admin-database-v1-2.0.2.jar
proto-google-cloud-bigtable-v2-1.16.1.jar
commons-lang-2.6.jar
gson-2.8.6.jar
google-oauth-client-1.31.2.jar
grpc-protobuf-lite-1.34.0.jar
beam-model-job-management-2.27.0.jar
google-cloud-core-grpc-1.93.9.jar
beam-runners-java-fn-execution-2.27.0.jar
gax-grpc-1.60.1.jar
netty-codec-4.1.51.Final.jar
failureaccess-1.0.1.jar
beam-sdks-java-expansion-service-2.27.0.jar
guava-retrying-2.0.0.jar
snappy-java-1.1.4.jar
grpc-google-cloud-pubsub-v1-1.90.3.jar
beam-sdks-java-io-google-cloud-platform-2.27.0.jar
jna-5.5.0.jar
google-api-client-1.30.10.jar
google-cloud-core-http-1.93.9.jar
hamcrest-2.1.jar
beam-model-pipeline-2.27.0.jar
kamon-core_2.12-2.1.4.jar
grpc-grpclb-1.34.0.jar
avro-1.8.2.jar
flogger-system-backend-0.5.1.jar
flogger-0.5.1.jar
proto-google-cloud-bigquerystorage-v1beta2-0.105.5.jar
beam-runners-direct-java-2.27.0.jar
simpleclient_pushgateway-0.9.0.jar
google-cloud-spanner-2.0.2.jar
google-http-client-protobuf-1.33.0.jar
okio-1.17.2.jar
netty-codec-socks-4.1.51.Final.jar
beam-vendor-guava-26_0-jre-0.1.jar
gax-httpjson-0.76.1.jar
beam-runners-google-cloud-dataflow-java-2.27.0.jar
netty-codec-http-4.1.51.Final.jar
simpleclient_common-0.9.0.jar
grpc-core-1.34.0.jar
google-http-client-appengine-1.36.0.jar
oshi-core-4.2.1.jar
httpcore-4.4.13.jar
javax.annotation-api-1.3.2.jar
google-api-services-iamcredentials-v1-rev20201022-1.30.10.jar
google-oauth-client-java6-1.31.2.jar
guava-30.0-jre.jar
commons-collections-3.2.2.jar
grpc-google-cloud-bigquerystorage-v1-1.5.5.jar
grpc-protobuf-1.34.0.jar
grpc-google-cloud-bigtable-v2-1.14.0.jar
jackson-core-2.12.0.jar
google-cloud-pubsublite-0.7.0.jar
commons-logging-1.2.jar
annotations-4.1.1.4.jar
google-api-services-dataflow-v1b3-rev20200713-1.30.10.jar
proto-google-cloud-pubsublite-v1-0.7.0.jar
vavr-match-0.10.2.jar
j2objc-annotations-1.3.jar
metrics-0.25.0.jar
proto-google-cloud-pubsub-v1-1.92.0.jarGoogle Cloud. A list of linked to the dataflow libraries: proto-google-cloud-spanner-v1-2.0.2.jar grpc-google-cloud-bigquerystorage-v1beta1-0.105.5.jar commons-io-2.6.jar grpc-alts-1.34.0.jar beam-runners-core-construction-java-2.27.0.jar beam-vendor-grpc-1_26_0-0.3.jar proto-google-cloud-spanner-admin-instance-v1-2.0.2.jar vavr-0.10.2.jar jackson-dataformat-yaml-2.12.0.jar grpc-api-1.34.0.jar jaxb-api-2.3.0.jar hamcrest-core-2.1.jar grpc-netty-1.32.2.jar kamon-bundle_2.12-2.1.4.jar beam-sdks-java-extensions-protobuf-2.27.0.jar grpc-netty-shaded-1.34.0.jar jackson-databind-2.12.0.jar api-common-1.10.1.jar paranamer-2.7.jar google-cloud-bigquery-1.122.2.jar google-auth-library-credentials-0.22.0.jar log4j-api-2.6.2.jar google-api-services-cloudresourcemanager-v1-rev20200720-1.30.10.jar proto-google-cloud-spanner-admin-database-v1-2.0.2.jar netty-codec-http2-4.1.51.Final.jar perfmark-api-0.19.0.jar netty-handler-4.1.51.Final.jar checker-compat-qual-2.5.5.jar beam-vendor-sdks-java-extensions-protobuf-2.27.0.jar beam-model-fn-execution-2.27.0.jar junit-4.13.1.jar google-api-services-bigquery-v2-rev20200916-1.30.10.jar netty-tcnative-boringssl-static-2.0.33.Final.jar resilience4j-core-1.6.1.jar kamon-prometheus_2.12-2.1.4.jar netty-common-4.1.51.Final.jar google-cloud-core-1.93.9.jar google-http-client-jackson2-1.38.0.jar grpc-google-cloud-spanner-v1-2.0.2.jar google-api-client-java6-1.30.10.jar proto-google-iam-v1-1.0.3.jar grpc-stub-1.34.0.jar classgraph-4.8.65.jar beam-sdks-java-fn-execution-2.27.0.jar commons-lang3-3.5.jar gax-1.60.1.jar proto-google-cloud-bigquerystorage-v1beta1-0.105.5.jar jsr305-3.0.2.jar grpc-google-cloud-bigtable-admin-v2-1.14.0.jar proto-google-cloud-bigquerystorage-v1alpha2-0.105.5.jar opencensus-api-0.24.0.jar beam-sdks-java-extensions-google-cloud-platform-core-2.27.0.jar opencensus-contrib-grpc-util-0.24.0.jar protobuf-java-util-3.14.0.jar netty-buffer-4.1.51.Final.jar google-api-services-pubsub-v1-rev20200713-1.30.10.jar httpclient-cache-4.5.9.jar google-api-services-storage-v1-rev20200927-1.30.10.jar google-auth-library-oauth2-http-0.22.0.jar proto-google-cloud-bigquerystorage-v1-1.5.5.jar simpleclient-0.9.0.jar auto-value-annotations-1.7.4.jar grpc-google-cloud-pubsublite-v1-0.7.0.jar metrics-core-3.2.6.jar listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar beam-sdks-java-core-2.27.0.jar google-cloud-bigquerystorage-1.5.5.jar error_prone_annotations-2.4.0.jar xz-1.8.jar protobuf-java-3.14.0.jar slf4j-api-1.7.30.jar grpc-auth-1.34.0.jar jna-platform-5.5.0.jar google-api-services-clouddebugger-v2-rev20200501-1.30.10.jar scala-library-2.12.11.jar datastore-v1-proto-client-1.6.3.jar joda-time-2.10.5.jar grpc-google-cloud-bigquerystorage-v1beta2-0.105.5.jar okhttp-3.14.7.jar google-api-services-healthcare-v1beta1-rev20200713-1.30.10.jar proto-google-common-protos-2.0.1.jar proto-google-cloud-bigtable-admin-v2-1.14.0.jar jackson-annotations-2.12.0.jar conscrypt-openjdk-uber-2.5.1.jar config-1.4.0.jar jackson-mapper-asl-1.9.13.jar commons-codec-1.14.jar grpc-context-1.34.0.jar args4j-2.33.jar commons-compress-1.20.jar json-20200518.jar grpc-google-cloud-spanner-admin-instance-v1-2.0.2.jar animal-sniffer-annotations-1.19.jar netty-handler-proxy-4.1.51.Final.jar logback-classic-1.2.3.jar google-api-client-jackson2-1.30.10.jar logstash-logback-encoder-6.2.jar logback-core-1.2.3.jar bigtable-client-core-1.16.0.jar threetenbp-1.5.0.jar snakeyaml-1.27.jar opencensus-contrib-http-util-0.24.0.jar google-http-client-1.38.0.jar google-cloud-pubsub-1.110.0.jar resilience4j-retry-1.6.1.jar gcsio-2.1.6.jar beam-vendor-bytebuddy-1_10_8-0.1.jar google-cloud-bigtable-1.14.0.jar proto-google-cloud-datastore-v1-0.88.0.jar grpc-google-common-protos-1.18.1.jar jackson-core-asl-1.9.13.jar util-2.1.6.jar checker-qual-3.7.0.jar google-extensions-0.5.1.jar httpclient-4.5.13.jar netty-resolver-4.1.51.Final.jar netty-transport-4.1.51.Final.jar grpc-google-cloud-spanner-admin-database-v1-2.0.2.jar proto-google-cloud-bigtable-v2-1.16.1.jar commons-lang-2.6.jar gson-2.8.6.jar google-oauth-client-1.31.2.jar grpc-protobuf-lite-1.34.0.jar beam-model-job-management-2.27.0.jar google-cloud-core-grpc-1.93.9.jar beam-runners-java-fn-execution-2.27.0.jar gax-grpc-1.60.1.jar netty-codec-4.1.51.Final.jar failureaccess-1.0.1.jar beam-sdks-java-expansion-service-2.27.0.jar guava-retrying-2.0.0.jar snappy-java-1.1.4.jar grpc-google-cloud-pubsub-v1-1.90.3.jar beam-sdks-java-io-google-cloud-platform-2.27.0.jar jna-5.5.0.jar google-api-client-1.30.10.jar google-cloud-core-http-1.93.9.jar hamcrest-2.1.jar beam-model-pipeline-2.27.0.jar kamon-core_2.12-2.1.4.jar grpc-grpclb-1.34.0.jar avro-1.8.2.jar flogger-system-backend-0.5.1.jar flogger-0.5.1.jar proto-google-cloud-bigquerystorage-v1beta2-0.105.5.jar beam-runners-direct-java-2.27.0.jar simpleclient_pushgateway-0.9.0.jar google-cloud-spanner-2.0.2.jar google-http-client-protobuf-1.33.0.jar okio-1.17.2.jar netty-codec-socks-4.1.51.Final.jar beam-vendor-guava-26_0-jre-0.1.jar gax-httpjson-0.76.1.jar beam-runners-google-cloud-dataflow-java-2.27.0.jar netty-codec-http-4.1.51.Final.jar simpleclient_common-0.9.0.jar grpc-core-1.34.0.jar google-http-client-appengine-1.36.0.jar oshi-core-4.2.1.jar httpcore-4.4.13.jar javax.annotation-api-1.3.2.jar google-api-services-iamcredentials-v1-rev20201022-1.30.10.jar google-oauth-client-java6-1.31.2.jar guava-30.0-jre.jar commons-collections-3.2.2.jar grpc-google-cloud-bigquerystorage-v1-1.5.5.jar grpc-protobuf-1.34.0.jar grpc-google-cloud-bigtable-v2-1.14.0.jar jackson-core-2.12.0.jar google-cloud-pubsublite-0.7.0.jar commons-logging-1.2.jar annotations-4.1.1.4.jar google-api-services-dataflow-v1b3-rev20200713-1.30.10.jar proto-google-cloud-pubsublite-v1-0.7.0.jar vavr-match-0.10.2.jar j2objc-annotations-1.3.jar metrics-0.25.0.jar proto-google-cloud-pubsub-v1-1.92.0.jar
Description
Hi ALL.
It seems there is some regression error in the last Beam version – 2.27.0.
There is a streaming dataflow that is runned in Google Cloud.
This dataflow is stable and has no any issues when built against Apache Beam version 2.23.0.
But if we build this dataflow against Apache Beam version 2.27.0 we catch an exception like below:
Error message from worker: java.lang.RuntimeException: java.lang.OutOfMemoryError: Direct buffer memory org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:932) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:972) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:195) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:149) Caused by: java.lang.OutOfMemoryError: Direct buffer memory java.base/java.nio.Bits.reserveMemory(Bits.java:175) java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) org.conscrypt.ConscryptEngineSocket$SSLInputStream.<init>(ConscryptEngineSocket.java:722) org.conscrypt.ConscryptEngineSocket.startHandshake(ConscryptEngineSocket.java:197) org.conscrypt.ConscryptEngineSocket.waitForHandshake(ConscryptEngineSocket.java:547) org.conscrypt.ConscryptEngineSocket.getOutputStream(ConscryptEngineSocket.java:290) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:476) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569) java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265) java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372) java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367) java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342) java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246) com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:113) com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$1(BigQueryServicesImpl.java:857) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1304) java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:834)
The error is rare. 99% of a time the dataflow works without this (actually any) error.
This issue blocks us to make upgrade of the Beam library for the dataflow.