Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35878

add fs.s3a.endpoint if unset and fs.s3a.endpoint.region is null

    XMLWordPrintableJSON

Details

    Description

      People working with S3A and hadoop 3.3.1 outside of EC2, and without the AWS CLI setup, are likely to hit: HADOOP-17771

      It should be straightforward to fix up the config similar to SPARK-35868...this will be backwards (harmless) and forwards compatible (it's the recommended workaround)

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-46793 Revert S3A endpoint fixup logic of SPARK-35878

          • Major - Major loss of function.
          • Closed
          relates to

          Sub-task - The sub-task of the issue HADOOP-17771 S3AFS creation fails "Unable to find a region via the region provider chain."

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-29250 Upgrade to Hadoop 3.3.1

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #33064 [Github] Pull Request #33064 (steveloughran)

          Web Link GitHub Pull Request #44834

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: