Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-12939

Improve default for IMPALA_BUILD_THREADS

    XMLWordPrintableJSON

Details

    • Task
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Infrastructure
    • None
    • ghx-label-8

    Description

      Improve the default selection for IMPALA_BUILD_THREADS and other parallelization config.

      Impala's build process needs 2GB of memory per CPU core with -notests or 4GB of memory per CPU core when building unit test binaries (the link process is especially memory-intensive). Exceeding these can lead to systems with many cores running out of memory during the build. We currently do not consider memory when selecting a default value for IMPALA_BUILD_THREADS.

      We currently default IMPALA_BUILD_THREADS to nproc, which also does not reflect CPU slicing that may happen in containers. How to detect this differs between cgroups v1 and v2 (v2 is becoming more common, present on Ubuntu 22 and RedHat 9).

      • v1:
        echo $(($(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us) / $(/sys/fs/cgroup/cpu/cpu.cfs_period_us)))
      • v2:
        awk '{ cores = ($1 == "max" ? '$(nproc)' : $1 / $2); print cores==int(cores) ? cores : int(cores)+1 }' /sys/fs/cgroup/cpu.max

      Something like the following should handle all cases

      if [[ -f /sys/fs/cgroup/cpu.max ]]; then
        awk '{ cores = ($1 == "max" ? '$(nproc)' : $1 / $2); print cores==int(cores) ? cores : int(cores)+1 }' /sys/fs/cgroup/cpu.max
      elif [[ -f /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]]; then
        echo $(($(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us) / $(/sys/fs/cgroup/cpu/cpu.cfs_period_us)))
      else
        nproc
      fi
      

      Attachments

        Activity

          People

            MikaelSmith Michael Smith
            MikaelSmith Michael Smith
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: