Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9461

S3Guard metadata store cannot be found

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • None
    • None
    • None
    • ghx-label-11

    Description

      I've seen this in a build on S3 + S3Guard. Many tests failed because S3Guard metadata store didn't exist:

      I0304 00:34:48.754648 17557 status.cc:126] couldn't retrieve FileSystem:
      DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2; auto-creation is turned off

       In impalad.ERROR: 

      hdfsBuilderConnect(forceNewInstance=0, nn=default, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error:
      ResourceNotFoundException: Requested resource not found: Table: impala-test-uswest2-1 not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: 6A5PINL6IDFF93BNDSSOQCG6JFVV4KQNSO5AEMVJF66Q9ASUAAJG)java.io.
      FileNotFoundException: DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2; auto-creation is turned off
         at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStoreTableManager.initTable(DynamoDBMetadataStoreTableManager.java:216)
         at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:427)
         at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:109)
         at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:418)
         at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3320)
      
      

      I suspected that the S3Guard metadata store initialization failed, so I checked load-test-warehouse-snapshot.log, but seems like S3Guard init was successful:

      20/03/03 22:03:36 INFO s3guard.S3GuardTool: Create metadata store: dynamodb://impala-test-uswest2-1 scheme: dynamodb
       20/03/03 22:03:38 INFO s3guard.S3GuardTool: Metadata store DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1, tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1} is initialized.
       Metadata Store Diagnostics:
         ARN=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1
         billing-mode=per-request
         description=S3Guard metadata store in DynamoDB
         name=impala-test-uswest2-1
         persist.authoritative.bit=true
         read-capacity=0
         region=us-west-2
         retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100 MILLISECONDS)
         size=9791493
         status=ACTIVE
         table={AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: impala-test-uswest2-1,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,                 CreationDateTime: Mon Mar 02 22:02:44 PST 2020,ProvisionedThroughput: {NumberOfDecreasesToday: 0,ReadCapacityUnits: 0,WriteCapacityUnits: 0},TableSizeBytes: 9791493,ItemCount: 47304,TableArn: arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1,TableId:       c0fad823-9fb7-48ba-b803-661a2b4052ee,BillingModeSummary: {BillingMode: PAY_PER_REQUEST,LastUpdateToPayPerRequestDateTime: Mon Mar 02 22:02:44 PST 2020},}
         write-capacity=0
       20/03/03 22:03:38 INFO s3guard.S3GuardTool: Create metadata store: dynamodb://impala-test-uswest2-1 scheme: dynamodb
       20/03/03 22:03:40 INFO s3guard.S3GuardTool: Metadata store DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1, tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1} is initialized.
       20/03/03 22:03:40 INFO s3guard.DynamoDBMetadataStore: Starting: Pruning DynamoDB Store
       20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Pruning DynamoDB Store: duration 1:38.538s
       20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Finished pruning 25981 items in batches of 25
      

      So based on this impala-test-uswest2-1 is successfully created in the region us-west-2, but later it became unreachable/deleted.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              boroknagyz Zoltán Borók-Nagy
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: