Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-8431

Consume all files in aws S3 bucket where deleteAfterRead = false

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.14.0
    • 2.15.4, 2.16.1, 2.17.0
    • camel-aws
    • None
    • Unknown

    Description

      The current AWS S3Consumer class has a problem if user supply the parameter deleteAfterRead=false. The S3Consumer will always consume the same files over and over again with the size of getMaxMessagesPerPoll().

      After some code chasing, i think, the root cause is because the ListObjectsRequest does not has the previous marker value. Hence, i wonder if we could do something like below:

      private string marker; // new line to define the marker string
      ..
      ..
      ..
      ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
      listObjectsRequest.setBucketName(bucketName);
      listObjectsRequest.setPrefix(getConfiguration().getPrefix());
      listObjectsRequest.setMaxKeys(maxMessagesPerPoll);
      listObjectsRequest.setMarker(marker); // new line to set the marker

      ObjectListing listObjects = getAmazonS3Client().listObjects(listObjectsRequest);
      marker = listObjects.getMarker(); // where marker is track

      Attachments

        Issue Links

          Activity

            People

              njiang Willem Jiang
              reusable Yap Poh Soon
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: