Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
Description
When starting jobServer with artifactDir pointing to a GCS bucket:
./gradlew :beam-runners-flink_2.11-job-server:runShadow -PflinkMasterUrl=localhost:8081 -PartifactsDir=gs://the-bucket
and running a Java portable pipeline with the following, portability related pipeline options:
--runner=PortableRunner --jobEndpoint=localhost:8099 --defaultEnvironmentType=DOCKER --defaultEnvironmentConfig=gcr.io/<my-freshly-built-sdk-harness-image>/java:latest'
I'm facing a series of OOM errors, like this:
Exception in thread "grpc-default-executor-3" java.lang.OutOfMemoryError: Java heap space at com.google.api.client.googleapis.media.MediaHttpUploader.buildContentChunk(MediaHttpUploader.java:606) at com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:408) at com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:336) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:508) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549) at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
This does not happen when I'm using a local filesystem for the artifact staging location.
Attachments
Attachments
Issue Links
- links to