Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-26677

Constrain available processors to Jetty during test runs to prevent thread exhaustion.

    XMLWordPrintableJSON

Details

    Description

      As described during a release candidate vote:

      HIVE-24484 introduced a change to limit hive.server2.webui.max.threads to 4. Jetty enforces thread leasing to warn or abort if there aren't enough threads available [1]. During startup, it attempts to lease a thread per NIO selector [2]. By default, the number of NIO selectors to use is determined based on available CPUs [3]. This is mostly a passthrough to Runtime.availableProcessors() [4]. In my case, running on a machine with 16 CPUs, this ended up creating more than 4 selectors, therefore requiring more than 4 threads and violating the lease check. I was able to work around this by passing the JETTY_AVAILABLE_PROCESSORS system property to constrain the number of CPUs available to Jetty.

      Since we are intentionally constraining the pool to 4 threads during itests, let's also limit JETTY_AVAILABLE_PROCESSORS in maven.test.jvm.args of the root pom.xml, so that others don't run into this problem later.

      [1] https://github.com/eclipse/jetty.project/blob/jetty-9.4.40.v20210413/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ThreadPoolBudget.java#L165
      [2] https://github.com/eclipse/jetty.project/blob/jetty-9.4.40.v20210413/jetty-io/src/main/java/org/eclipse/jetty/io/SelectorManager.java#L255
      [3] https://github.com/eclipse/jetty.project/blob/jetty-9.4.40.v20210413/jetty-io/src/main/java/org/eclipse/jetty/io/SelectorManager.java#L79
      [4] https://github.com/eclipse/jetty.project/blob/jetty-9.4.40.v20210413/jetty-util/src/main/java/org/eclipse/jetty/util/ProcessorUtils.java#L45

      Attachments

        Issue Links

          Activity

            People

              cnauroth Chris Nauroth
              cnauroth Chris Nauroth
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h