Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-12596

bulkload needs to follow locality

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.98.8
    • 0.98.14, 1.3.0, 2.0.0
    • HFile, regionserver
    • None
    • hadoop-2.3.0, hbase-0.98.8, jdk1.7

    • Reviewed

    Description

      Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the "withFavoredNodes" method, and we just need to call it in HFileOutputFormat's getNewWriter().
      This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it.

      Attachments

        1. HBASE-12596-branch-1-v2.patch
          19 kB
          Victor Xu
        2. HBASE-12596-branch-1-v1.patch
          19 kB
          Victor Xu
        3. HBASE-12596-0.98-v6.patch
          18 kB
          Victor Xu
        4. HBASE-12596-master-v6.patch
          17 kB
          Victor Xu
        5. HBASE-12596-master-v5.patch
          15 kB
          Victor Xu
        6. HBASE-12596-0.98-v5.patch
          15 kB
          Victor Xu
        7. HBASE-12596-master-v4.patch
          11 kB
          Victor Xu
        8. HBASE-12596-0.98-v4.patch
          10 kB
          Victor Xu
        9. HBASE-12596-master-v3.patch
          11 kB
          Victor Xu
        10. HBASE-12596-0.98-v3.patch
          10 kB
          Victor Xu
        11. HBASE-12596-master-v2.patch
          6 kB
          Victor Xu
        12. HBASE-12596-0.98-v2.patch
          6 kB
          Victor Xu
        13. HBASE-12596-master-v1.patch
          7 kB
          Victor Xu
        14. HBASE-12596-0.98-v1.patch
          6 kB
          Victor Xu
        15. HBASE-12596.patch
          5 kB
          Victor Xu

        Activity

          People

            victorunique Victor Xu
            victorunique Victor Xu
            Votes:
            0 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: