Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21354

Lock The Entire Table If Majority Of Partitions Are Locked

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.0, 4.0.0
    • None
    • HiveServer2
    • None

    Description

      One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism. When a Hive query interacts with a table which has a lot of partitions, this may put a lot of stress on the ZK system.

      Please add a heuristic that works like this:

      1. Count the number of partitions that a query is required to lock
      2. Obtain the total number of partitions in the table
      3. If the number of partitions accessed by the query is greater than or equal to half the total number of partitions, simply create one ZNode lock at the table level.

      This would improve performance of many queries, but in particular, a select count(1) from table ... or ... select * from table limit 5 where the table has many partitions.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            belugabehr David Mollitor Assign to me
            belugabehr David Mollitor

            Dates

              Created:
              Updated:

              Slack

                Issue deployment