Details
-
Task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
Hadoop hosts have had their ulimit -u raised from 10000 to 30000 (per user, by INFRA). The Docker build container has had its limit raised from 10000 to 12500.
Description
Seeing this kind of thing in nightly...
java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread at org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) Caused by: java.lang.OutOfMemoryError: unable to create new native thread at org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing our docker container. It is set to 20G (The hosts are 48G). Fork count is 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add the maven 4G and we could be over the top.
Play w/ downing the fork size (in earlier study we didn't seem to need this much RAM when running a fat long test). Let me also take th ms off the mvn allocation to see if that helps.
Attachments
Attachments
Issue Links
- relates to
-
-
- Resolved
-
- links to
1.
|
Get ulimit and how many threads we can make up on hadoop nodes |
|
Resolved | Unassigned |
2.
|
Change fork count from 0.5C to 0.25C |
|
Resolved | Michael Stack |
3.
|
Up the container nproc uplimit from 10000 to 12500 |
|
Resolved | Michael Stack |
4.
|
Temporary issue... ignore |
|
Resolved | Unassigned |