Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
  3. HADOOP-13028

add low level counter metrics for S3A; use in read performance tests

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.0
    • 2.8.0, 3.0.0-alpha1
    • fs/s3, metrics
    • None
    • Reviewed

    Description

      against S3 (and other object stores), opening connections can be expensive, closing connections may be expensive (a sign of a regression).

      S3A FS and individual input streams should have counters of the # of open/close/failure+reconnect operations, timers of how long things take. This can be used downstream to measure efficiency of the code (how often connections are being made), connection reliability, etc.

      Attachments

        1. HADOOP-13028-001.patch
          43 kB
          Steve Loughran
        2. HADOOP-13028-002.patch
          64 kB
          Steve Loughran
        3. HADOOP-13028-004.patch
          77 kB
          Steve Loughran
        4. HADOOP-13028-005.patch
          101 kB
          Steve Loughran
        5. HADOOP-13028-006.patch
          90 kB
          Steve Loughran
        6. HADOOP-13028-007.patch
          94 kB
          Steve Loughran
        7. HADOOP-13028-008.patch
          106 kB
          Steve Loughran
        8. HADOOP-13028-009.patch
          110 kB
          Steve Loughran
        9. HADOOP-13028-012.patch
          128 kB
          Steve Loughran
        10. HADOOP-13028-013.patch
          128 kB
          Steve Loughran
        11. HADOOP-13028-branch-2-008.patch
          106 kB
          Steve Loughran
        12. HADOOP-13028-branch-2-009.patch
          110 kB
          Steve Loughran
        13. HADOOP-13028-branch-2-010.patch
          128 kB
          Steve Loughran
        14. HADOOP-13028-branch-2-011.patch
          130 kB
          Steve Loughran
        15. HADOOP-13028-branch-2-012.patch
          130 kB
          Steve Loughran
        16. HADOOP-13028-branch-2-013.patch
          130 kB
          Steve Loughran
        17. org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
          94 kB
          Steve Loughran
        18. org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
          176 kB
          Steve Loughran

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: