Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18477 Über-jira: S3A Hadoop 3.3.9 features
  3. HADOOP-18844

Add a way to get the IOStatistics of active filesystems in long-lived processes

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.3.9
    • None
    • fs/azure, fs/s3
    • None

    Description

      we can configure abfs and s3a to print out the filesystem IOStats in close(), but this doesn't let us see the state of long-lived processes, including latencies, error rates, and other important information.

      Proposed: we add a means by which all cached filesystems which provide IOStatistics can log them to a file/console while the process continues to run

      • a configurable scheduled thread which enumerates all open filesystem instances, gets their iostats and if non empty dumps to a log...log configuration can set that to the console or elsewhere. Good: can run without intervention; bad: one more thread to manage.
      • option to add a signal handler which will do this whenever a SIGUSR2 is raised. JVM signal handling is a bit of a brittle feature, so it should be off by default.
        org.apache.hadoop.service.launcher.IrqHandler supports signal wire-up. good: gives us an equivalent of kill -quit to get a view of iostats. bad: you need to be on the machine

      Attachments

        Activity

          People

            Unassigned Unassigned
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: