CPU Benchmark

Configuration

• tf-nightly-gpu (20180814)
  • pip install tf-nightly-gpu==1.11.0.dev20180814
• Benchmark code: Based on TF's benchmarks (GitHub hash 8f95262).

Notice:

• The default calculation of params.datasets_num_private_threads is problematic: when CPU has few cores datasets_num_private_threads can be set to 1 and this severely affects the speed. As a hacked-up solution we set num_monitoring_threads=0 in benchmark_cnn.py, so datasets_num_private_threads = num_cpu_thread - 2 * num_gpus.
• Alexnet (large parameter, low computation) causes significant bottleneck in real images training across all CPUs. Not be able to reproduce TF's reference Alexnet results on training real images. (7900x is reasonablly close without distortions)
• Threadripper causes undeterministic segfault. For the successful runs it is on par with i7-7820X
  • Solved by using higher speed RAM.
• Threadripeer CPU (1950X and 2990WX) use 128GB 2666 MHz RAM.
• xeon CPUs has some issues with intensive pre-processing (extremely poor performance on Alexnet)
• E5-2686 is tested with 4 V100 GPUs (AWS P3.8xlarge instance)
• all_reduce_spec is not fully tested (needs NCCL2).

Instructions

Prepare Imagenet data. Here is a link to download a fraction of imagenet. Download and unzip it into ~/data/imagenet_mini

cd scripts/tf_cnn_benchmarks
./cpu_benchmark.sh
./cpu_report.sh

The results will be saved as summary.md.

Results

syn-parameter_server-4gpus

syn-replicated-4gpus

real-parameter_server-distortions-4gpus

real-parameter_server-4gpus

real-replicated-distortions-4gpus

real-replicated-4gpus