Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19326

stats auto gather: incorrect aggregation during UNION queries (may lead to incorrect results)

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • Statistics
    • None

    Description

      Found when investigating the results change after converting tables to MM, turns out the MM result is correct but the current one is not.
      The test ends like so:

      desc formatted small_alltypesorc_a;
      ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
      desc formatted small_alltypesorc_a;
      insert into table small_alltypesorc_a select * from small_alltypesorc1a;
      desc formatted small_alltypesorc_a;
      

      The results from the descs in the golden file are:

      	COLUMN_STATS_ACCURATE	{\"BASIC_STATS\":\"true\"}
      	numFiles            	1                   
      	numRows             	5                               
      ...
      	COLUMN_STATS_ACCURATE	{\"BASIC_STATS\":\"true\"}
      	numFiles            	1                   
      	numRows             	15                                
      ...
      	COLUMN_STATS_ACCURATE	{\"BASIC_STATS\":\"true\"}
      	numFiles            	2                   
      	numRows             	20                              
      

      Note the result change after analyze - the original nomRows is inaccurate, but BASIC_STATS is set to true.

      I am assuming with metadata only optimization this can produce incorrect results.

      Attachments

        1. HIVE-19326.13.patch
          91 kB
          Zoltan Haindrich
        2. HIVE-19326.13.patch
          91 kB
          Zoltan Haindrich
        3. HIVE-19326.12.patch
          89 kB
          Zoltan Haindrich
        4. HIVE-19326.11.patch
          89 kB
          Zoltan Haindrich
        5. HIVE-19326.11.patch
          89 kB
          Zoltan Haindrich
        6. HIVE-19326.10.patch
          90 kB
          Zoltan Haindrich
        7. HIVE-19326.09.patch
          90 kB
          Zoltan Haindrich
        8. HIVE-19326.08.patch
          89 kB
          Zoltan Haindrich
        9. HIVE-19326.07.patch
          89 kB
          Zoltan Haindrich
        10. HIVE-19326.06wip05.patch
          1.02 MB
          Zoltan Haindrich
        11. HIVE-19326.06wip04.patch
          1.02 MB
          Zoltan Haindrich
        12. HIVE-19326.06wip03.patch
          1.82 MB
          Zoltan Haindrich
        13. HIVE-19326.06wip02.patch
          1.02 MB
          Zoltan Haindrich
        14. HIVE-19326.06wip01.patch
          175 kB
          Zoltan Haindrich
        15. HIVE-19326.06.patch
          89 kB
          Zoltan Haindrich
        16. HIVE-19326.05.patch
          80 kB
          Zoltan Haindrich
        17. HIVE-19326.04.patch
          76 kB
          Zoltan Haindrich
        18. HIVE-19326.03.patch
          58 kB
          Zoltan Haindrich
        19. HIVE-19326.02.patch
          56 kB
          Zoltan Haindrich
        20. HIVE-19326.01wip01.patch
          17 kB
          Zoltan Haindrich

        Issue Links

          Activity

            People

              kgyrtkirk Zoltan Haindrich
              sershe Sergey Shelukhin
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: