Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18517

limit max log message width in log4j

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.5.0, 2.0.0-alpha-2
    • 1.5.0, 2.0.0-alpha-2, 2.0.0
    • None
    • None
    • Incompatible change, Reviewed
    • Sets a log length max of 1000 characters.

    Description

      We had two cases now in our prod / pilot setups which is leading to humongous log lines in RegionServer logs.

      In first case, one of the phoenix user had constructed a query with a really large list of Id filters (61 MB) that translated into HBase scan that was running slow which lead to responseTooSlow messages in the logs with the entire filter list being printed in the logs, example
      ipc.RpcServer - (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1501457864417,"responsesize":11,"method":"Scan","param":"region { type: REGION_NAME value: ..... org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter\\022\351\\200\\036
      n(org.apache.phoenix.filter.SkipScanFilter ... <a really long filter list leading to single log lines of size 61MB> ...

      There was another case where a use case had created a table with really large key/region names. This was causing humongous log lines for flush and compaction on these regions filling up the RS logs

      These large logs usually cause issues with disk I/O load, loading the splunk servers, even machine perf degradations. With 61 MB log lines basic log processing commands like vim, scrolling the logs, wc -l , etc were getting stuck. High GC activity was also noted on this cluster although not 100% sure if it was related to above issue.

      We should consider limiting the message size in logs which can be easily done by adding a maximum width format modifier on the message conversion character in log4j.properties
      log4j.appender.console.layout.ConversionPattern=...: %m%n
      to
      log4j.appender.console.layout.ConversionPattern=...: %.10000m%n

      Attachments

        1. HBASE-18517.branch-1.001.patch
          2 kB
          Vikas Vishwakarma
        2. HBASE-18517.master.001.patch
          2 kB
          Vikas Vishwakarma

        Activity

          People

            vik.karma Vikas Vishwakarma
            vik.karma Vikas Vishwakarma
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: