Details
-
Task
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
ghx-label-8
Description
Improve the default selection for IMPALA_BUILD_THREADS and other parallelization config.
Impala's build process needs 2GB of memory per CPU core with -notests or 4GB of memory per CPU core when building unit test binaries (the link process is especially memory-intensive). Exceeding these can lead to systems with many cores running out of memory during the build. We currently do not consider memory when selecting a default value for IMPALA_BUILD_THREADS.
We currently default IMPALA_BUILD_THREADS to nproc, which also does not reflect CPU slicing that may happen in containers. How to detect this differs between cgroups v1 and v2 (v2 is becoming more common, present on Ubuntu 22 and RedHat 9).
- v1:
echo $(($(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us) / $(/sys/fs/cgroup/cpu/cpu.cfs_period_us)))
- v2:
awk '{ cores = ($1 == "max" ? '$(nproc)' : $1 / $2); print cores==int(cores) ? cores : int(cores)+1 }' /sys/fs/cgroup/cpu.max
Something like the following should handle all cases
if [[ -f /sys/fs/cgroup/cpu.max ]]; then awk '{ cores = ($1 == "max" ? '$(nproc)' : $1 / $2); print cores==int(cores) ? cores : int(cores)+1 }' /sys/fs/cgroup/cpu.max elif [[ -f /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]]; then echo $(($(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us) / $(/sys/fs/cgroup/cpu/cpu.cfs_period_us))) else nproc fi