Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-23956

Use less resources running tests

    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-alpha-1, 2.3.0
    • test
    • None
    • Reviewed

    Description

      Our tests can create thousands of threads all up in the one JVM. Using less means less memory, less contention, likelier passes, and later, more possible parallelism.

      I've been studying the likes of TestNamespaceReplicationWithBulkLoadedData to see what it does as it runs (this test puts up 4 clusters with replication between). It peaks at 2k threads. After some configuration and using less HDFS, its possible to get it down to ~800 threads and about 1/2 the memory-used. HDFS is a main offender. DataXceivers (Server and Client), jetty threads, Volume threads (async disk 'worker' then another for cleanup...), image savers, ipc clients – new thread per incoming connection w/o bound (or reuse), block responder threads, anonymous threads, and so on. Many are not configurable or boundable or are hard-coded; e.g. each volume gets 4 workers regardless. Biggest impact was just downing the count of data nodes. TODO: a follow-on that turns down DN counts in all tests.

      I've been using Java Flight Recorder during this study. Here is how you get a flight recorder for the a single test run: {code:java} MAVEN_OPTS=" -XX:StartFlightRecording=disk=true,dumponexit=true,filename=recording.jfr,settings=profile,path-to-gc-roots=true,maxsize=1024m" mvn test -Dtest=TestNamespaceReplicationWithBulkLoadedData -Dsurefire.firstPartForkCount=0 -Dsurefire.secondPartForkCount=0 {code} i.e. start recording on mvn launch, bound the size of the recording, and have the test run in the mvn context (DON'T fork). Useful is connecting to the running test at the same time from JDK Mission Control. We do the latter because the thread reporting screen is overwhelmed by the count of running threads and if you connect live, you can at least get a 'live threads' graph w/ count as the test progresses. Useful. When the test finishes, it dumps a .jfr file which can be opened in JDK MC.

      I've been compiling w/ JDK8 and then running w/ JDK11 so I can use JDK MC Version 7, the non-commercial latest. Works pretty well. Let me put up a patch for tests that cuts down thread counts where we can.

      Let me put up a patch that does first pass on curtailing resource usage.

      Attachments

        1. Screen Shot 2020-03-09 at 10.11.35 PM.png
          91 kB
          Michael Stack
        2. Screen Shot 2020-03-09 at 10.11.26 PM.png
          63 kB
          Michael Stack
        3. Screen Shot 2020-03-10 at 7.10.15 AM.png
          629 kB
          Michael Stack
        4. Screen Shot 2020-03-10 at 7.10.28 AM.png
          342 kB
          Michael Stack
        5. Screen Shot 2020-03-09 at 10.20.08 PM.png
          91 kB
          Michael Stack

        Issue Links

          Activity

            People

              stack Michael Stack
              stack Michael Stack
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: