Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7926

long-lived daemons for query fragment execution, I/O and caching

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • None
    • Hide
      LLAP is the new hybrid execution model that enables efficiencies across queries, such as caching of columnar data, JIT-friendly operator pipelines, and reduced overhead for multiple queries (including concurrent queries), as well as new performance features like asynchronous I/O, pre-fetching and multi-threaded processing. The hybrid model consists of a long-lived service interacting with on-demand elastic containers serving as a tightly integrated DAG-based framework for query execution.

      The first version of LLAP is being shipped in Hive 2.0 release. The component has been extensively exercised on test and live clusters, and tested, but is expected to have rough edges in this initial release.
      The current limitations are: supported with Tez only; does not support ACID tables; the I/O elevator and cache only support ORC format and vectorized execution.
      Show
      LLAP is the new hybrid execution model that enables efficiencies across queries, such as caching of columnar data, JIT-friendly operator pipelines, and reduced overhead for multiple queries (including concurrent queries), as well as new performance features like asynchronous I/O, pre-fetching and multi-threaded processing. The hybrid model consists of a long-lived service interacting with on-demand elastic containers serving as a tightly integrated DAG-based framework for query execution. The first version of LLAP is being shipped in Hive 2.0 release. The component has been extensively exercised on test and live clusters, and tested, but is expected to have rough edges in this initial release. The current limitations are: supported with Tez only; does not support ACID tables; the I/O elevator and cache only support ORC format and vectorized execution.

    Description

      We are proposing a new execution model for Hive that is a combination of existing process-based tasks and long-lived daemons running on worker nodes. These nodes can take care of efficient I/O, caching and query fragment execution, while heavy lifting like most joins, ordering, etc. can be handled by tasks.
      The proposed model is not a 2-system solution for small and large queries; neither it is a separate execution engine like MR or Tez. It can be used by any Hive execution engine, if support is added; in future even external products (e.g. Pig) can use it.

      The document with high-level design we are proposing will be attached shortly.

      Attachments

        1. LLAPdesigndocument.pdf
          242 kB
          Sergey Shelukhin
        1.
        		 LLAP: separate decoding thread from read/uncompress thread Sub-task Open Sergey Shelukhin
        2.
        		 LLAP: implement pause and stop for async data production Sub-task Open Unassigned
        3.
        		 LLAP: row-level vectorized SARGs Sub-task Patch Available Yohei Abe
        4.
        		 LLAP: (IO) consider using reactive framework to string together parts of code Sub-task Open Unassigned
        5.
        		 LLAP: ORC decoding of row groups for complex types Sub-task Open Prasanth Jayachandran
        6.
        		 LLAP: consider specialized "transient" metadata cache Sub-task Open Unassigned
        7.
        		 documentation for llap Sub-task Open Gunther Hagleitner
        8.
        		 LLAP: Add rack based scheduling of work Sub-task Open Unassigned
        9.
        		 fix yarn service registry not found in ut problem Sub-task Open Gunther Hagleitner
        10.
        		 LLAP : add LLAP IO read debug tool Sub-task Open Sergey Shelukhin
        11.
        		 LLAP: adjust allocation after decompression Sub-task Open Unassigned
        12.
        		 LLAP: enable yourkit profiling of tasks Sub-task Patch Available Sergey Shelukhin
        13.
        		 LLAP: Use task number, attempt number to cache plans Sub-task Open Unassigned
        14.
        		 LLAP: Tez heartbeats are delayed by ~500+ ms due to Hadoop IPC client Sub-task Open Siddharth Seth
        15.
        		 LLAP: general cache deadlock avoidance Sub-task Open Sergey Shelukhin
        16.
        		 Move fragment execution onto a thread pool Sub-task Open Unassigned
        17.
        		 LLAP: Make use of additional information to determine run/preemption order Sub-task Open Unassigned
        18.
        		 LLAP: fix container sizing configuration for memory Sub-task Open Vikram Dixit K
        19.
        		 LLAP: allocator occasionally has a spurious failure to allocate due to "partitioned" locking and has to retry Sub-task Open Sergey Shelukhin
        20.
        		 LLAP: Add counters for time lost per query due to preemption Sub-task Open Unassigned
        21.
        		 LLAP: task scheduler thread-count keeps growing Sub-task Open Siddharth Seth
        22.
        		 LLAP: Exception reported while trying to kill a task Sub-task Open Prasanth Jayachandran
        23.
        		 LLAP: Better handling of hostnames when sending heartbeats to the AM Sub-task Open Unassigned
        24.
        		 LLAP: Serialize handling of requests / events for a query within daemons Sub-task Open Siddharth Seth
        25.
        		 LLAP: make ObjectCache for plans work properly in the daemon Sub-task Patch Available Sergey Shelukhin
        26.
        		 Create LLAP Monitor Daemon class and launch scripts Sub-task Open Yuya OZAWA

        Activity

          People

            sershe Sergey Shelukhin
            sershe Sergey Shelukhin
            Votes:
            3 Vote for this issue
            Watchers:
            47 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: