Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"

Beam: 2.31.0

Flink 1.12.0

Python 3.8.5

# tr -dc "A-Za-z 0-9" < /dev/urandom | fold -w100|head -n 9000000 > bigfile.txt

# python3 wordcount_with_metrics.py --input beam/bigfile.txt --out bigfile-output.txt

 

 

E0603 04:50:42.552403830    8442 chttp2_transport.cc:1117]   Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"

ERROR:apache_beam.runners.worker.data_plane:Failed to read inputs in the data plane.

Traceback (most recent call last):

  File "/usr/local/lib/python3.8/dist-packages/apache_beam/runners/worker/data_plane.py", line 581, in _read_inputs

    for elements in elements_iterator:

  File "/usr/local/lib/python3.8/dist-packages/grpc/channel.py", line 426, in __next_

    return self._next()

  File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 826, in _next

    raise self

grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:

status = StatusCode.UNAVAILABLE

details = "Socket closed"

debug_error_string = "{"created":"@1622695842.552861174","description":"Error received from peer ipv6:[::1]:45717","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"Socket closed","grpc_status":14}"

>

Exception in thread read_grpc_client_inputs:

Traceback (most recent call last):

  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner

    self.run()

  File "/usr/lib/python3.8/threading.py", line 870, in run

    self._target(*self._args, **self._kwargs)

  File "/usr/local/lib/python3.8/dist-packages/apache_beam/runners/worker/data_plane.py", line 598, in <lambda>

    target=lambda: self._read_inputs(elements_iterator),

  File "/usr/local/lib/python3.8/dist-packages/apache_beam/runners/worker/data_plane.py", line 581, in _read_inputs

    for elements in elements_iterator:

  File "/usr/local/lib/python3.8/dist-packages/grpc/channel.py", line 426, in __next_

    return self._next()

  File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 826, in _next

    raise self

grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:

status = StatusCode.UNAVAILABLE

details = "Socket closed"

debug_error_string = "{"created":"@1622695842.552861174","description":"Error received from peer ipv6:[::1]:45717","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"Socket closed","grpc_status":14}"