XMLWordPrintableJSON

Details

    Description

      parquet, avro etc are still stuck building with older hadoop releases.

      This makes using new APIs hard (PARQUET-2171) and means that APIs which are 5 years old such as HADOOP-15229 just aren't picked up.

      This lack of openFIle() adoption hurts working with files in cloud storage as

      • extra HEAD requests are made
      • read policies can't be explicitly set
      • split start/end can't be passed down

      Proposed

      1. create class org.apache.hadoop.io.WrappedOperations
      2. add methods to wrap the apis
      3. test in contract tests via reflection loading -verifies we have done it properly.

      Attachments

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: