Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-483

PERFORMANCE: different strategies for large and small order bys

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.2.0
    • None
    • None

    Description

      Currently pig always does a multi-pass order by where it first determines a distribution for the keys and then orders in a second pass. This avoids the necessity of having a single reducer. However, in cases where the data is small enough to fit into a single reducer, this is inefficient. For small data sets it would be good to realize the small size of the set and do the order by in a single pass with a single reducer.

      This is a candidate project for Google summer of code 2011. More information about the program can be found at http://wiki.apache.org/pig/GSoc2011

      Attachments

        1. PIG-483.0.patch
          6 kB
          Jie Li
        2. PIG-483.1.patch
          10 kB
          Jie Li

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. PIG-2779 Refactoring the code for setting number of reducers

          • Major - Major loss of function.
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. PIG-2675 Optimization: Remove unnecessary Limit jobs from plan

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. PIG-2772 Convert the skew join to the normal join for small dataset

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. PIG-2784 Framework for dynamic query optimization

          • Major - Major loss of function.
          • Open

          Activity

            People

              jay23jack Jie Li
              olgan Olga Natkovich
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated: