Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-15364

Add support for S3 Select to S3A

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0
    • fs/s3
    • None

    Description

      Expect a PoC patch for this in a couple of days;

      • it'll depend on an SDK update to work, plus a couple of of other minor changes
      • Adds command line option too 
        hadoop s3guard select -header use -compression gzip -limit 100 s3a://landsat-pds/scene_list.gz" \
"SELECT s.entityId FROM S3OBJECT s WHERE s.cloudCover = '0.0' "

      For wider use we'll need to implement the HADOOP-15229 so that callers can pass down the expression along with any other parameters

      Attachments

        1. HADOOP-15364-001.patch
          90 kB
          Steve Loughran
        2. HADOOP-15364-002.patch
          91 kB
          Steve Loughran
        3. HADOOP-15364-004.patch
          88 kB
          Steve Loughran

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail"

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-15229 Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15606 FSDataInputStream/FSInputStream to implement getFileLength()

          • Major - Major loss of function.
          • Open
          is depended upon by

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-7182 MapReduce input format/record readers to support S3 select queries

          • Major - Major loss of function.
          • Resolved
          is part of

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-15229 Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: