Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4207

Provide approximate number of input records to be processed in UnorderedKVInput

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.10.0
    • None
    • None

    Description

      There are cases when broadcasted data is loaded into hashtable in upstream applications (e.g Hive). Apps tends to predict the number of entries in the hashtable diligently, but there are cases where these estimates can be very complicated at compile time.

       

      Tez can help in such cases, by providing "approximate number of input records counter", to be processed in UnorderedKVInput. This is to avoid expensive rehash when hashtable sizes are not estimated correctly. It would be good to start with broadcast first and then to move on to unordered partitioned case later.

       

      This would help in predicting the number of entries at runtime & can get better estimates for hashtable.

      Attachments

        1. TEZ-4207.1.patch
          7 kB
          Rajesh Balamohan
        2. TEZ-4207.wip.patch
          8 kB
          Rajesh Balamohan

        Activity

          People

            rajesh.balamohan Rajesh Balamohan
            rajesh.balamohan Rajesh Balamohan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: