Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17566 Über-jira: S3A Hadoop 3.3.2 features
  3. HADOOP-17511

Add an Audit plugin point for S3A auditing/context

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.1
    • 3.3.2
    • fs/s3

    Description

      Add a way for auditing tools to correlate S3 object calls with Hadoop FS API calls.

      Initially just to log/forward to an auditing service.

      Later: let us attach them as parameters in S3 requests, such as opentrace headeers or (my initial idea: http referrer header -where it will get into the log)

      Challenges

      • ensuring the audit span is created for every public entry point. That will have to include those used in s3guard tools, some defacto public APIs
      • and not re-entered for active spans. s3A code must not call back into the FS API points
      • Propagation across worker threads

      Documentation Links

      Attachments

        Issue Links

          causes

          Sub-task - The sub-task of the issue HADOOP-17822 fs.s3a.acl.default not working after S3A Audit feature added

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-18091 S3A auditing leaks memory through ThreadLocal references

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-18094 Disable S3A auditing by default.

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          contains

          Sub-task - The sub-task of the issue HADOOP-16911 S3A mkdirs to indicate which parent path element refers to a file

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-17470 Collect more S3A IOStatistics

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-16613 s3a to set fake directory marker contentType to application/x-directory

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-17380 ITestS3AContractSeek.teardown closes FS before superclass does its cleanup

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          incorporates

          Sub-task - The sub-task of the issue HADOOP-17498 Add a MkdirOperation for chained S3 operations during mkdir

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-14661 S3A to support Requester Pays Buckets

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-18659 ABFS to add Auditing support and attach http referrer to requests

          • Major - Major loss of function.
          • Open
          links to

          Web Link GitHub Pull Request #2675

          Web Link GitHub Pull Request #2807

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 22h 20m
                  22h 20m