Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-12596

bulkload needs to follow locality

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.98.8
    • Fix Version/s: 0.98.14, 1.3.0, 2.0.0
    • Component/s: HFile, regionserver
    • Labels:
      None
    • Environment:

      hadoop-2.3.0, hbase-0.98.8, jdk1.7

    • Hadoop Flags:
      Reviewed

      Description

      Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the "withFavoredNodes" method, and we just need to call it in HFileOutputFormat's getNewWriter().
      This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it.

        Attachments

        1. HBASE-12596-master-v6.patch
          17 kB
          Victor Xu
        2. HBASE-12596-master-v5.patch
          15 kB
          Victor Xu
        3. HBASE-12596-master-v4.patch
          11 kB
          Victor Xu
        4. HBASE-12596-master-v3.patch
          11 kB
          Victor Xu
        5. HBASE-12596-master-v2.patch
          6 kB
          Victor Xu
        6. HBASE-12596-master-v1.patch
          7 kB
          Victor Xu
        7. HBASE-12596-branch-1-v2.patch
          19 kB
          Victor Xu
        8. HBASE-12596-branch-1-v1.patch
          19 kB
          Victor Xu
        9. HBASE-12596-0.98-v6.patch
          18 kB
          Victor Xu
        10. HBASE-12596-0.98-v5.patch
          15 kB
          Victor Xu
        11. HBASE-12596-0.98-v4.patch
          10 kB
          Victor Xu
        12. HBASE-12596-0.98-v3.patch
          10 kB
          Victor Xu
        13. HBASE-12596-0.98-v2.patch
          6 kB
          Victor Xu
        14. HBASE-12596-0.98-v1.patch
          6 kB
          Victor Xu
        15. HBASE-12596.patch
          5 kB
          Victor Xu

          Activity

            People

            • Assignee:
              victorunique Victor Xu
              Reporter:
              victorunique Victor Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: