This is NeuPIMs cycle-accurate simulator. We integrate open-source NPU simulator and PIM simulator to implement NeuPIMs simulator.
- Paper: https://dl.acm.org/doi/10.1145/3620666.3651380
- Authors: Guseul Heo, Sangyeop Lee, Jaehong Cho, Hyunmin Choi, and Sanghyeon Lee (KAIST); Hyungkyu Ham and Gwangsun Kim (POSTECH); Divya Mahajan (Georgia Tech); Jongse Park (KAIST)
- torch >= 1.10.1
- conan == 1.57.0
- onnxruntime >= 1.10.0
- cmake >= 3.22.1 (You need to build manually)
- gcc == 8.3
$ git clone https://github.com/casys-kaist/NeuPIMs.git
$ cd ai-framwork-sim
$ docker build . -t neupims
build docker image
$ docker run -it neupims
(docker) cd ai-framwork-sim
(docker) ./brun.sh
run docker image
$ git clone https://github.com/casys-kaist/NeuPIMs.git
$ cd ai-framwork-sim
$ git submodule update --recursive --init
$ mkdir build && cd build
$ conan install .. --build missing
$ cmake ..
$ make -j
$ cd ..
$ ./brun.sh
- NPU-only: Codes on
npu-onlybranch, all operations in LLM batched inference are executed on NPU. - NPU+PIM: Codes on
npu+pimbranch, attention GEMV operations on PIM. PIM is single row buffered PIM on this baseline. - NeuPIMs: Codes on
mainbranch, we use dual row buffer PIM and sub-batch interleaving technique. (Sub-batch interleaving is enabled only when batch size>=256)
If you use NeuPIMs for your research, please cite our paper:
@inproceedings{10.1145/3620666.3651380,
author = {Heo, Guseul and Lee, Sangyeop and Cho, Jaehong and Choi, Hyunmin and Lee, Sanghyeon and Ham, Hyungkyu and Kim, Gwangsun and Mahajan, Divya and Park, Jongse},
title = {NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing},
year = {2024},
doi = {10.1145/3620666.3651380},
booktitle = {Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3},
series = {ASPLOS '24}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent