Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Resolved
-
None
Description
Multiple Beam PostCommits seem to be failing due to a variety of errors, such as "503 Service unavailable", "429 Too Many Requests", or "404 Not Found". It's hard to add a complete description of the issue here since there are too many tests failing. Also made worse because some other flakes and issues seem to be masking these failures. But I'll try to find a few examples:
https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3661/
11:17:01 BadStatusCodeError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>: response: <{'status': '429', 'content-length': '598', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Wed, 26 Jun 2019 18:17:01 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json; charset=UTF-8'}>, content <{
11:17:01 "error": {
11:17:01 "code": 429,
11:17:01 "message": "Quota exceeded for quota metric 'dataflow.googleapis.com/create_requests' and limit 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for consumer 'project_number:844138762903'.",
11:17:01 "status": "RESOURCE_EXHAUSTED",
11:17:01 "details": [
11:17:01 {
11:17:01 "@type": "type.googleapis.com/google.rpc.Help",
11:17:01 "links": [
11:17:01 {
11:17:01 "description": "Google developer console API key",
11:17:01 "url": "https://console.developers.google.com/project/844138762903/apiui/credential"
11:17:01 }
11:17:01 ]
11:17:01 }
11:17:01 ]
11:17:01 }
11:17:01 }
https://builds.apache.org/job/beam_PostCommit_Python_Verify/8598/
BadStatusCodeError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>: response: <{'status': '429', 'content-length': '598', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'se
rver': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Wed, 26 Jun 2019 18:10:42 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json; charset=UTF-8'}>, content <{
"error": {
"code": 429,
"message": "Quota exceeded for quota metric 'dataflow.googleapis.com/create_requests' and limit 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for consumer 'project_number:844138762903'.",
"status": "RESOURCE_EXHAUSTED",
"details": [
{
"@type": "type.googleapis.com/google.rpc.Help",
"links": [
{
"description": "Google developer console API key",
"url": "https://console.developers.google.com/project/844138762903/apiui/credential"
}
https://builds.apache.org/job/beam_PostCommit_Java/3617/
java.lang.RuntimeException : Failed to create a workflow job: The service is currently unavailable.
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException:
503 Service Unavailable
{
"code" : 503,
"errors" : [ {
"domain" : "global",
"message" : "The service is currently unavailable.",
"reason" : "backendError"
} ],
"message" : "The service is currently unavailable.",
"status" : "UNAVAILABLE"
}
Dataflow SDK version: 2.15.0-SNAPSHOT