Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22306

Use nonblocking thrift server for metastore

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Metastore
    • None

    Description

      Currently hive metastore's threads are blocking for network io (it's using `TThreadPoolServer` behind the scene), which means with increasing use cases (in our tech stack there are different services relying on it, hiveserver2, spark, presto, and more, all with a significant number of users), to handle all connections it needs either a big thread pool or many instances with smaller thread pools. And often, those metastores will see their thread pool saturated, while the cpu usage is still quite low, just because most connections stay idle and only run a query from time to time. This is thus a great misuse of the computation resources.

      Thus I propose to use a non blocking threading model, and run computation asynchronously. 

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              q.xu Qinghui Xu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: