Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14946

Don't allow multi's to over run the max result size.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.2.0, 1.3.0, 2.0.0
    • 1.2.0, 1.3.0, 2.0.0
    • Client, IPC/RPC
    • None
    • Reviewed
    • Hide
      The HBase region server will now send a chunk of get responses to a client if the total response size is too large. This will only be done for clients 1.2.0 and beyond. Older clients by default will have the old behavior.

      This patch is for the case where the basic flow is like this:

      I want to get a single column from lots of rows. So I create a list of gets. Then I send them to table.get(List<Get>). If the regions for that table are spread out then those requests get chunked out to all the region servers. No one regionserver gets too many. However if one region server contains lots of regions for that table then a multi action can contain lots of gets. No single get is too onerous. However the regionserver won't return until every get is complete. So if there are thousands of gets that are sent in one multi then the regionserver can retain lots of data in one thread.
      Show
      The HBase region server will now send a chunk of get responses to a client if the total response size is too large. This will only be done for clients 1.2.0 and beyond. Older clients by default will have the old behavior. This patch is for the case where the basic flow is like this: I want to get a single column from lots of rows. So I create a list of gets. Then I send them to table.get(List<Get>). If the regions for that table are spread out then those requests get chunked out to all the region servers. No one regionserver gets too many. However if one region server contains lots of regions for that table then a multi action can contain lots of gets. No single get is too onerous. However the regionserver won't return until every get is complete. So if there are thousands of gets that are sent in one multi then the regionserver can retain lots of data in one thread.

    Description

      If a user puts a list of tons of different gets into a table we will then send them along in a multi. The server un-wraps each get in the multi. While no single get may be over the size limit the total might be.

      We should protect the server from this.
      We should batch up on the server side so each RPC is smaller.

      Attachments

        1. 14946.add.missing.annotations.addendum.patch
          2 kB
          Michael Stack
        2. HBASE-14946.patch
          5 kB
          Elliott Neil Clark
        3. HBASE-14946-v1.patch
          12 kB
          Elliott Neil Clark
        4. HBASE-14946-v10.patch
          45 kB
          Elliott Neil Clark
        5. HBASE-14946-v11.patch
          49 kB
          Elliott Neil Clark
        6. HBASE-14946-v2.patch
          7 kB
          Elliott Neil Clark
        7. HBASE-14946-v3.patch
          8 kB
          Elliott Neil Clark
        8. HBASE-14946-v5.patch
          14 kB
          Elliott Neil Clark
        9. HBASE-14946-v6.patch
          40 kB
          Elliott Neil Clark
        10. HBASE-14946-v7.patch
          41 kB
          Elliott Neil Clark
        11. HBASE-14946-v8.patch
          47 kB
          Elliott Neil Clark
        12. HBASE-14946-v9.patch
          45 kB
          Elliott Neil Clark

        Activity

          People

            eclark Elliott Neil Clark
            eclark Elliott Neil Clark
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: