Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0
    • fs/azure
    • None

    Description

      When interacting with Azure Blob Storage via the Hadoop FileSystem client, it's currently (as of a8bbd81) not possible to set custom metadata attributes.

      Here is a snippet that demonstrates the missing behavior (throws an UnsupportedOperationException):

      val blobAccount = "SET ME"
      val blobKey = "SET ME"
      val blobContainer = "SET ME"
      val blobFile = "SET ME"
      
      import org.apache.hadoop.conf.Configuration
      import org.apache.hadoop.fs.{FileSystem, Path}
      
      val conf = new Configuration()
      conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
      conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
      
      val path = new Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
      
      val fs = FileSystem.get(path, conf)
      fs.setXAttr(path, "somekey", "somevalue".getBytes)
      

      Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem inherits the default setXAttr from FileSystem which throws the UnsupportedOperationException.

      The underlying Azure Blob Storage service does support custom metadata (service docs) as does the azure-storage SDK that's being used by NativeAzureFileSystem (SDK docs).

      Is there another way that I should be setting custom metadata on Azure Blob Storage files? Is there a specific reason why setXAttr hasn't been implemented on NativeAzureFileSystem? If not, I can take a shot at implementing it.

      Attachments

        Activity

          People

            c-w Clemens Wolff
            c-w Clemens Wolff
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: