Scale compute power about llvm-premerge-checks HOT 6 CLOSED

setting up VM for testing:

sudo apt-get update; sudo apt-get install -y --no-install-recommends cmake ninja-build git ca-certificates clang-7 lld-7  clang ccache python python3 build-essential python-psutil arcanist zip wget python3-pip python3-setuptools
git clone --depth=1 https://github.com/llvm/llvm-project.git
cd llvm-project
git clone https://github.com/google/llvm-premerge-checks.git
mkdir build
cd build
export CC=clang-7
export CXX=clang++-7
export LD=LLD
export CCACHE_PATH=~/ccache
mkdir -p ${CCACHE_PATH}
cmake -GNinja ../llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_LLD=ON  -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libcxx;libcxxabi;lld" 	-DLLVM_CCACHE_BUILD=ON -DLLVM_CCACHE_DIR="${CCACHE_PATH}" -DLLVM_CCACHE_MAXSIZE=20G     -DLLVM_ENABLE_ASSERTIONS=ON -DCMAKE_CXX_FLAGS=-gmlt     -DLLVM_LIT_ARGS="-v --xunit-xml-output ${WORKSPACE}/build/test-results.xml"

# clean build
time ninja all
# incremental build
rm -rf *
cmake ... #as before
time ninja all
time ninja check
time ninja check-all

from llvm-premerge-checks.

• benchmark results: https://docs.google.com/spreadsheets/d/1ArXlUQYUbdumUXQakWKSbeo1ttMRP4QcoiAf_7-fB3M/
• 600 commits / week : https://github.com/llvm/llvm-project/graphs/commit-activity
• about 5 commit/hour on average

my conclusions (from the numbers so far):

• compiling scales linearly with number of cores (to be expected)
• testing scales sub-linear with number of cores (not sure why that is).
• a local SSD is a major speedup for ccache compared to a networked SSD
• from 16 to 32 cores we will get a significant speedup
• from 32 to 64 cores the speedup is less impressive as testing does not scale that well
• bootstraping the ccache would avoid the slow first builds
• the prices scale linear with number of cores

decision:

• use local SSD
• use 32 cores per machine (for mix of clean and incremental builds)
• look into bootstrapping ccache
• look into moving to C2 machines
• assuming build times of 20 minutes and over-provisioning of 1.5:
  • we will start with 3x32 cores

from llvm-premerge-checks.

Testing is sublinear because of Amdahl's Law. In short, as more cores are added, the time to complete the test suite equals the time to complete the longest test. Adding more cores means fixing slow tests or relegating them to a bot that only tests "long tests" (Swift has this development model).

from llvm-premerge-checks.

In short, as more cores are added, the time to complete the test suite equals the time to complete the longest test.

This applies equally to the building phase though right? (which suffers from the link being a bottleneck frequently)

from llvm-premerge-checks.

Yup. That being said, developers are more motivated to fix slow compiles or links than slow tests.

from llvm-premerge-checks.

For example:
llvm/llvm-project@d488daa
llvm/llvm-project@d15f319

from llvm-premerge-checks.