Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-372

Nested UDFs cause _very_ high memory usage when processing query

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.3.0
    • Query Processor, UDF
    • None
    • Fedora Linux, 10x Amazon EC2 (Large Instance w/ 8GB Ram)

    • Reviewed

    Description

      When nesting UDFs, the Hive Query processor takes a large amount of time+memory to process the query. For example, I ran something along the lines of:

      select trim( trim( trim(trim( trim( trim( trim( trim( trim(column))))))))) from test_table;

      This query needs 10GB+ of memory to process before it'll launch the job. The amount of memory increases exponentially with each nested UDF.

      Obviously, I am using trim() in this case as a simple example that causes the same problem to occur. In my actual use-case I had a bunch of nested regexp_replaces.

      Attachments

        1. HIVE-372.2.trunk.patch
          6 kB
          Zheng Shao
        2. HIVE-372.2.patch
          6 kB
          Zheng Shao
        3. HIVE-372.1.patch
          5 kB
          Zheng Shao

        Issue Links

          Activity

            People

              zshao Zheng Shao
              scorona Steve Corona
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: