Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15309

GZIP HFile Compression & Decompression Memory Leak, Similar Issue HBASE-5516

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Not A Bug
    • Affects Version/s: 0.94.6
    • Fix Version/s: 0.94.6
    • Component/s: HFile
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I'd memory leak issue in regionserver process, VM and RSS memory continiously increasing 64MB.
      I'm using hbase-0.94.6 , MALLOC_ARENA_MAX=4 was set in hbase-env.sh
      I've huge write load and frequent minor compaction, We've used GZip hfile compression.
      Max java regionserver heap size is 32GB.

      top - 14:28:30 up 201 days, 21:06,  3 users,  load average: 5.67, 3.72, 3.31
      Tasks: 803 total,   1 running, 802 sleeping,   0 stopped,   0 zombie
      Cpu(s):  8.3%us,  2.1%sy,  0.0%ni, 85.9%id,  3.5%wa,  0.0%hi,  0.1%si,  0.0%st
      Mem:  65932340k total, 63961912k used,  1970428k free,  2394528k buffers
      Swap: 29659132k total,    63532k used, 29595600k free,  1095268k cached
      
       PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
       57335 hbase     20   0 46.4g  44g 9296 S 98.2 70.9  13319:10 java
      
       [hbase@xxxxxx-hslave ~]$  pmap -x 57335 | sort -k 3 -nr | more
      total kB        48695984 46765792 46756512
      00007ff312460000 33171072 33169464 33169464 rwx--    [ anon ]
      000000004010a000 1448552 1448552 1448552 rwx--    [ anon ]
      00007ff2d1810000  612120  603124  603124 rwx--    [ anon ]
      00007ff2fadff000  383364  383364  383364 rwx--    [ anon ]
      00007ff0e8000000  131072  131072  131072 rwx--    [ anon ]
      00007ff218000000  131048  131048  131048 rwx--    [ anon ]
      00007ff128000000  131068  131048  131048 rwx--    [ anon ]
      00007ff230000000  131044  131044  131044 rwx--    [ anon ]
      00007ff000000000  131036  131036  131036 rwx--    [ anon ]
      00007fefe0000000  131060  131036  131036 rwx--    [ anon ]
      00007ff23c000000   65536   65536   65536 rwx--    [ anon ]
      00007ff0a4000000   65536   65536   65536 rwx--    [ anon ]
      00007ff054000000   65536   65536   65536 rwx--    [ anon ]
      00007ff01c000000   65536   65536   65536 rwx--    [ anon ]
      00007fefb4000000   65536   65536   65536 rwx--    [ anon ]
      00007ff22c000000   65532   65532   65532 rwx--    [ anon ]
      00007ff110000000   65532   65532   65532 rwx--    [ anon ]
      00007ff10c000000   65532   65532   65532 rwx--    [ anon ]
      00007ff0b8000000   65532   65532   65532 rwx--    [ anon ]
      00007ff09c000000   65532   65532   65532 rwx--    [ anon ]
      00007feff8000000   65532   65532   65532 rwx--    [ anon ]
      00007ff250000000   65528   65528   65528 rwx--    [ anon ]
      --More--
      
      $pmap -x 57335 | awk '{print $3}' | awk '{ if($i<65536 && $i>64000) print $i}' | wc -l
      146
      

      Regionserver process has many anon pages, size varying from 64000 to 65536 KB that shown above.

        Attachments

          Activity

            People

            • Assignee:
              tlipcon Todd Lipcon
              Reporter:
              karthikshva123 karthikp
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved: