Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18477 Über-jira: S3A Hadoop 3.3.9 features
  3. HADOOP-17961

s3 and abfs incremental listing: use SAX parsers to stream results to list iterators

Add voteVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.3.2
    • None
    • fs/azure, fs/s3
    • None

    Description

      With code gradually adopting listStatusIncremental(), asking for a smaller initial batch could permit faster ramp of result processing.

      probably most significant on an s3 versioned bucket, as there the need to skip tombstones can result in significantly slower listings -but could benefit ABFS too

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            stevel@apache.org Steve Loughran

            Dates

              Created:
              Updated:

              Slack

                Issue deployment