Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-14747

S3AInputStream to implement CanUnbuffer

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.1
    • 3.3.0
    • fs/s3
    • None

    Description

      HBase relies on FileSystems implementing CanUnbuffer.unbuffer() to force input streams to free up remote connections (HBASE-9393). This works for HDFS, but not elsewhere.

      S3A input stream can implement CanUnbuffer.unbuffer() by closing the input stream and relying on lazy seek to reopen it on demand.

      Needs

      • Contract specification of unbuffer. As in "who added a new feature to filesystems but forgot to mention what it should do?"
      • Contract test for filesystems which declare their support.
      • S3AInputStream to call closeStream() on a call to unbuffer().
      • Test case

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HADOOP-14748 Wasb input streams to implement CanUnbuffer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14872 CryptoInputStream should implement unbuffer

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12805 Annotate CanUnbuffer with @InterfaceAudience.Public

          • Major - Major loss of function.
          • Closed
          relates to

          Sub-task - The sub-task of the issue HADOOP-14765 AdlFsInputStream should implement unbuffer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HBASE-9393 Region Server fails to properly close socket resulting in many CLOSE_WAIT to Data Nodes

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-7694 FSDataInputStream should support "unbuffer"

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #690

          Activity

            People

              stakiar Sahil Takiar
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: