Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Cannot Reproduce
-
None
-
None
-
None
-
ghx-label-11
Description
I've seen this in a build on S3 + S3Guard. Many tests failed because S3Guard metadata store didn't exist:
I0304 00:34:48.754648 17557 status.cc:126] couldn't retrieve FileSystem: DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2; auto-creation is turned off
In impalad.ERROR:
hdfsBuilderConnect(forceNewInstance=0, nn=default, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error: ResourceNotFoundException: Requested resource not found: Table: impala-test-uswest2-1 not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: 6A5PINL6IDFF93BNDSSOQCG6JFVV4KQNSO5AEMVJF66Q9ASUAAJG)java.io. FileNotFoundException: DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2; auto-creation is turned off at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStoreTableManager.initTable(DynamoDBMetadataStoreTableManager.java:216) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:427) at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:109) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:418) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3320)
I suspected that the S3Guard metadata store initialization failed, so I checked load-test-warehouse-snapshot.log, but seems like S3Guard init was successful:
20/03/03 22:03:36 INFO s3guard.S3GuardTool: Create metadata store: dynamodb://impala-test-uswest2-1 scheme: dynamodb 20/03/03 22:03:38 INFO s3guard.S3GuardTool: Metadata store DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1, tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1} is initialized. Metadata Store Diagnostics: ARN=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1 billing-mode=per-request description=S3Guard metadata store in DynamoDB name=impala-test-uswest2-1 persist.authoritative.bit=true read-capacity=0 region=us-west-2 retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100 MILLISECONDS) size=9791493 status=ACTIVE table={AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: impala-test-uswest2-1,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE, CreationDateTime: Mon Mar 02 22:02:44 PST 2020,ProvisionedThroughput: {NumberOfDecreasesToday: 0,ReadCapacityUnits: 0,WriteCapacityUnits: 0},TableSizeBytes: 9791493,ItemCount: 47304,TableArn: arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1,TableId: c0fad823-9fb7-48ba-b803-661a2b4052ee,BillingModeSummary: {BillingMode: PAY_PER_REQUEST,LastUpdateToPayPerRequestDateTime: Mon Mar 02 22:02:44 PST 2020},} write-capacity=0 20/03/03 22:03:38 INFO s3guard.S3GuardTool: Create metadata store: dynamodb://impala-test-uswest2-1 scheme: dynamodb 20/03/03 22:03:40 INFO s3guard.S3GuardTool: Metadata store DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1, tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1} is initialized. 20/03/03 22:03:40 INFO s3guard.DynamoDBMetadataStore: Starting: Pruning DynamoDB Store 20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Pruning DynamoDB Store: duration 1:38.538s 20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Finished pruning 25981 items in batches of 25
So based on this impala-test-uswest2-1 is successfully created in the region us-west-2, but later it became unreachable/deleted.
Attachments
Issue Links
- relates to
-
IMPALA-8754 S3 with S3Guard tests encounter "ResourceNotFoundException" from DynamoDB
- Resolved