Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18027

Replication should respect RPC size limits when batching edits

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.4.0, 1.3.1, 2.0.0
    • 1.4.0, 1.3.2, 2.0.0
    • Replication
    • None
    • Reviewed

    Description

      In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in batches. We create N lists. N is the minimum of configured replicator threads, number of 100-waledit batches, or number of current sinks. Every pending entry in the replication context is then placed in order by hash of encoded region name into one of these N lists. Each of the N lists is then sent all at once in one replication RPC. We do not test if the sum of data in each N list will exceed RPC size limits. This code presumes each individual edit is reasonably small. Not checking for aggregate size while assembling the lists into RPCs is an oversight and can lead to replication failure when that assumption is violated.

      We can fix this by generating as many replication RPC calls as we need to drain a list, keeping each RPC under limit, instead of assuming the whole list will fit in one.

      Attachments

        1. HBASE-18027-branch-1.patch
          30 kB
          Andrew Kyle Purtell
        2. HBASE-18027.patch
          31 kB
          Andrew Kyle Purtell
        3. HBASE-18027-branch-1.patch
          29 kB
          Andrew Kyle Purtell
        4. HBASE-18027.patch
          30 kB
          Andrew Kyle Purtell
        5. HBASE-18027-branch-1.patch
          29 kB
          Andrew Kyle Purtell
        6. HBASE-18027.patch
          30 kB
          Andrew Kyle Purtell
        7. HBASE-18027-branch-1.patch
          10 kB
          Andrew Kyle Purtell
        8. HBASE-18027.patch
          14 kB
          Andrew Kyle Purtell
        9. HBASE-18027.patch
          14 kB
          Andrew Kyle Purtell
        10. HBASE-18027.patch
          14 kB
          Andrew Kyle Purtell
        11. HBASE-18027.patch
          30 kB
          Andrew Kyle Purtell
        12. HBASE-18027.patch
          29 kB
          Andrew Kyle Purtell

        Issue Links

          Activity

            People

              apurtell Andrew Kyle Purtell
              apurtell Andrew Kyle Purtell
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: