Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-10103

Add WasbIO / WasbFileSystem (Azure Blob Storage)

Details

    • New Feature
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • io-ideas
    • None

    Description

      Azure Blob Storage can be accessed by using the wasb:// and wasbs:// protocols. This should be quite similar to the hdfs:// implementations already there.

       

      We should just be able to use it like this:

       

              (
                  p
                  | ReadFromText("yourcontainer@youraccount.blob.core.windows.net/test/sample.txt")
                  | WriteToText("output.txt", shard_name_template="", num_shards=0)
              )
      

       

       

      See:

       [1] https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-use-blob-storage

       [2] https://hadoop.apache.org/docs/current/hadoop-azure/index.html

       [3] https://gerardnico.com/azure/wasb

      Attachments

        Activity

          People

            Unassigned Unassigned
            epicfaace Ashwin Ramaswami
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: