Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-28144

Remove overly verbose debug messages from MetastoreDirectSqlUtils

    XMLWordPrintableJSON

Details

    Description

      When BITVECTOR or KLL stats are disabled/not present in the metastore the following message may appear way too often in the HMS logs. 

      2024-03-22T01:50:57,849 DEBUG [CachedStore-CacheUpdateService: Thread-240] metastore.MetastoreDirectSqlUtils: Expected blob type but got java.lang.String
      

      In fact in some cases, the message appears more than once for every single partition that is present in the table(s) being queried. When the number of partitions is important it can easily clog the logs with redundant and useless information.

      To put things in perspective while running the cbo_query10.q on the statistics of TPC-DS30TB dataset the message occupies more than 50% (26MB) of the total log file (46MB).

      $mvn test -Dtest=TestTezTPCDS30TBPerfCliDriver -Dqfile=cbo_query10.q
      $grep -a "Expected blob type but got java.lang.String" target/tmp/log/hive.log | wc -c
      26129538
      $ wc -c target/tmp/log/hive.log 
      46959003 target/tmp/log/hive.log
      

      The presence of the message does not tells us much on its own. In conjunction with the code we can infer that we are not fetching BITVECTOR/KLL stats from the metastore but this could be done in a different place without having to print the same message 170K times.
       
      Removing this message saves disk space, avoids frequent log rotation, and improves the overall readability of the log file.

      There is another redundant
      message which appears when transforming a database value to Boolean. The message is redundant since it is followed directly by an exception so there is no reason to have both. This message may not appear as often as the previous one but given that it doesn't add much value it can also be removed.

      Attachments

        Issue Links

          Activity

            People

              zabetak Stamatis Zampetakis
              zabetak Stamatis Zampetakis
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: