Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-11815

Fail to read more than 1M of items with DynamoDBIO

Details

    • Bug
    • Status: Resolved
    • P1
    • Resolution: Fixed
    • None
    • 2.29.0
    • io-java-aws
    • None

    Description

      In processElement:

      private static class ReadFn<T> extends DoFn<Read<T>, T> {
        @ProcessElement
        public void processElement(@Element Read<T> spec, OutputReceiver<T> out) {
          AmazonDynamoDB client = spec.getAwsClientsProvider().createDynamoDB();
          ScanRequest scanRequest = spec.getScanRequestFn().apply(null);
          scanRequest.setSegment(spec.getSegmentId());
          ScanResult scanResult = client.scan(scanRequest);
          out.output(spec.getScanResultMapperFn().apply(scanResult));
        }
      }
      

       
       

      if total requested items > 1M dynamodb will only send upto 1M of data  and expects client to read remaining items through LastEvaluatedKey (see this)

       

      Attachments

        Issue Links

          Activity

            People

              aromanenko Alexey Romanenko
              Safari Mohammad
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m