Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-23779

Up the default fork count to make builds complete faster; make count relative to CPU count

VotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-alpha-1, 2.3.0
    • test
    • None
    • Reviewed
    • Hide
      Pass --threads=2 building on jenkins. It shortens nightly build times by about ~25%.

      It works by running module build/test in parallel when dependencies allow. Upping the forkcount beyond the pom default of 0.25C would have us broach our CPU budget on jenkins when two modules are running in parallel (2 modules at 0.25% of CPU each makes 0.5C and on jenkins, hadoop nodes run two jenkins executors per host). Higher forkcounts also seems to threaten build stability.

      For running tests locally, to go faster, up fork count.

      $ x="0.5C" ; mvn --threads=2 -Dsurefire.firstPartForkCount=$x -Dsurefire.secondPartForkCount=$x test -PrunAllTests

      You could up the x from 0.5C to 1.0C but YMMV (On overcommitted hardware, tests start bombing out pretty soon after startup). You could try upping thread count but on occasion are likely to overcommit hardware.
      Show
      Pass --threads=2 building on jenkins. It shortens nightly build times by about ~25%. It works by running module build/test in parallel when dependencies allow. Upping the forkcount beyond the pom default of 0.25C would have us broach our CPU budget on jenkins when two modules are running in parallel (2 modules at 0.25% of CPU each makes 0.5C and on jenkins, hadoop nodes run two jenkins executors per host). Higher forkcounts also seems to threaten build stability. For running tests locally, to go faster, up fork count. $ x="0.5C" ; mvn --threads=2 -Dsurefire.firstPartForkCount=$x -Dsurefire.secondPartForkCount=$x test -PrunAllTests You could up the x from 0.5C to 1.0C but YMMV (On overcommitted hardware, tests start bombing out pretty soon after startup). You could try upping thread count but on occasion are likely to overcommit hardware.

    Description

      Tests take a long time. Our fork count running all tests are conservative – 1 (small) for first part and 5 for second part (medium and large). Rather than hardcoding we should set the fork count to be relative to machine size. Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box.

      Looking up at jenkins, it seems like the boxes are 24 cores... at least going by my random survey. The load reported on a few seems low though this not representative (looking at machine/uptime).

      More parallelism willl probably mean more test failure. Let me take a look see.

      Attachments

        1. addendum2.patch
          2 kB
          Michael Stack
        2. Screen Shot 2020-04-08 at 9.19.53 AM.png
          386 kB
          Michael Stack
        3. Screen Shot 2020-04-13 at 4.09.49 PM.png
          179 kB
          Michael Stack
        4. test_yetus_934.0.patch
          0.2 kB
          Nick Dimiduk

        Issue Links

        breaks

        Bug - A problem which impairs or prevents the functions of the product. HBASE-23923 Set suitable forkCount when running pre commit

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved
        is related to

        Test - A new unit, integration or system test. HBASE-23783 Address tests writing and reading SSL/Security files in a common location.

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Wish - General wishlist item. HBASE-23795 Enable all tests to be run in parallel on reused JVMs.

        • Major - Major loss of function.
        • Resolved

        Task - A task that needs to be done. HBASE-24072 Nightlies reporting OutOfMemoryError: unable to create new native thread

        • Major - Major loss of function.
        • Resolved
        relates to

        Bug - A problem which impairs or prevents the functions of the product. YETUS-934 Precommit does not accept a caller's value of MAVEN_ARGS

        • Major - Major loss of function.
        • Resolved
        links to

        Web Link GitHub Pull Request #1108

        Web Link GitHub Pull Request #1461

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            stack Michael Stack
            stack Michael Stack
            Votes:
            0 Vote for this issue
            Watchers:
            8 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment