Instructions on benching AI model with anonymous compiler

Docker

use following cmd to install ipex and intel ai model zoo.

source setup_env.sh

Please refer to Simplified Bert Large Inference. The detailed guidance is in Detailed Bert Large Inference

Please refer to Simplified DLRM Inference. The detailed guidance is in Detailed DLRM Inference

By default, only int8 mode of IPEX will run into oneDNN Graph. If you want to benchmark the performance under fp32 mode with oneDNN Graph and Anonymous Compiler, you need to manually do some modification on IPEX and Model Zoo source code. For IPEX, navigate to [IPEX_ROOT]/csrc/jit/codegen/onednn/interface.cpp and change the value of llga_fp32_bf16_enabled from false to true:

bool llga_fp32_bf16_enabled = true;

Then re-build IPEX:

export DNNL_GRAPH_BUILD_COMPILER_BACKEND=1
python setup.py install

For Model Zoo Scripts, you need to search for the model’s main script and comment out the ipex.optimize in the corresponding dtype’s condition branch. For instance, as for DLRM model under fp32 mode, navigate to dlrm_s_pytorch.py, comment out the following code:

dlrm = ipex.optimize(dlrm, dtype=torch.float, ...)

Note that, the above changes need to be reverted when you are running in int8 mode. As for benchmark AI models, you can follow the instructions on Bert Large and DLRM to prepare the model specified dependency and datasets. For bench DLRM model, you need to modify the below two args to switch to throughput mode (32 cores per instance).

OMP_NUM_THREADS=1 => OMP_NUM_THREADS=32
-share-weight-instance=$Cores => -share-weight-instance=0

By default, anonymous compiler is enabled on IPEX. To get oneDNN's performance data on end-to-end model, you need to set the below environment variable:

export _ONEDNN_GRAPH_DISABLE_COMPILER_BACKEND=1

To check whether the Anonymous Compiler is enabled or not, you can check the graph verbose (via export ONEDNN GRAPH VERBOSE=1). You will see the below outputs from verbose once compiler backend is enabled.

onednn_graph_verbose,info,backend,0:compiler_backend
onednn_graph_verbose,info,backend,1:dnnl_backend