Uploaded image for project: 'Apache Knox'
  1. Apache Knox
  2. KNOX-2954

Gateway service metric name contain hbase rowkey led to frequently full gc

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Patch Available
    • Critical
    • Resolution: Unresolved
    • None
    • None
    • Metrics, Server
    • None

    Description

      Our Gateway Server frequently full gc in a few days. I dumped jvm heap and analyzed dump file with mat tool. I found some leak suspects about jmx metric, These leaks occurs in class org.apache.knox.gateway.services.metrics.impl.DefaultMetricsService. 

      This is the analysis result generated by MAT

       

      This is detail of problem Suspect 3

       

      I found that many metrics which starting with service contain hbase rowkey by using visualVM tool. These metrics were generted when I used get api to query data from hbase. I think that name of service metric should not contain rowkey.

       

      This is an example of metric name, the query command is 

      curl -ik -u xx:xx   \
       -H "Accept: text/xml"  \
       -X GET \
       'https://localhost:8443/gateway/sandbox/hbase/demo/row[1-9]' 

       

      The reason for this problem is that client and service metric both call the InstrUtils.getServiceResourcePath function to get resource path , but the parameters they use are different when build metric name. Suppose the request url is https://localhost:8443/gateway/sandbox/hbase/demo/row9. When building a client metric name, the parameter of the calling InstrUtils.getServiceResourcePath function is httpServletRequest.getPathInfo(), httpServletRequest.getPathInfo() value is /hbase/demo/row9. When building a service metric name, the parameter of the calling InstrUtils.getServiceResourcePath function is RequestLine.getUri(), RequestLine.getUri() value is /demo/row9. The purpose of InstrUtils.getServiceResourcePath is to obtain the first two elements of path. So service metric name contains rowkey.

      I think that we should only obtain the first elements of path, when building service metric name.

      The effect after applying the  pr791

       

       

       

       

       

       

       

      Attachments

        1. image-2023-09-01-17-37-51-597.png
          43 kB
          liang.feng
        2. image-2023-09-01-16-55-30-378.png
          46 kB
          liang.feng
        3. image-2023-09-01-16-49-57-346.png
          127 kB
          liang.feng
        4. image-2023-09-01-16-40-35-876.png
          127 kB
          liang.feng
        5. mat_heap_leak_suspects_result.png
          151 kB
          liang.feng

        Issue Links

          Activity

            People

              Unassigned Unassigned
              a516072575 liang.feng
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: