Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18072 Über-JIRA: abfs phase III: Hadoop 3.4.0 features & fixes
  3. HADOOP-18876

ABFS: Change default from disk to bytebuffer for fs.azure.data.blocks.buffer

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.6
    • 3.4.0
    • build
    • Reviewed
    • Hide
      The default value for fs.azure.data.blocks.buffer is changed from "disk" to "bytebuffer"

      This will speed up writing to azure storage, at the risk of running out of memory
      -especially if there are many threads writing to abfs at the same time and the
      upload bandwidth is limited.

      If jobs do run out of memory writing to abfs, change the option back to "disk"
      Show
      The default value for fs.azure.data.blocks.buffer is changed from "disk" to "bytebuffer" This will speed up writing to azure storage, at the risk of running out of memory -especially if there are many threads writing to abfs at the same time and the upload bandwidth is limited. If jobs do run out of memory writing to abfs, change the option back to "disk"

    Description

      Change default from disk to bytebuffer for fs.azure.data.blocks.buffer.

      Gathered from multiple workload runs, the presented data underscores a noteworthy enhancement in performance. The adoption of ByteBuffer for reading operations exhibited a remarkable improvement of approximately 64.83% when compared to traditional disk-based reading. Similarly, the implementation of ByteBuffer for write operations yielded a substantial efficiency gain of about 60.75%. These findings underscore the consistent and substantial advantages of integrating ByteBuffer across a range of workload scenarios.

      Attachments

        Issue Links

          Activity

            People

              asrani_anmol Anmol Asrani
              asrani_anmol Anmol Asrani
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: