Code Monkey home page Code Monkey logo

neupims's Introduction

NeuPIMs Simulator

This is NeuPIMs cycle-accurate simulator. We integrate open-source NPU simulator and our in-house PIM simulator based on DRAMsim3 to implement NeuPIMs simulator.

Publication

  • Paper: https://dl.acm.org/doi/10.1145/3620666.3651380
  • Authors: Guseul Heo, Sangyeop Lee, Jaehong Cho, Hyunmin Choi, and Sanghyeon Lee (KAIST); Hyungkyu Ham and Gwangsun Kim (POSTECH); Divya Mahajan (Georgia Tech); Jongse Park (KAIST)

Python Package

  • torch >= 1.10.1
  • conan == 1.57.0
  • onnxruntime >= 1.10.0

Package

  • cmake >= 3.22.1 (You need to build manually)
  • gcc == 8.3

Getting Started

method 1 (Docker Image)

$ git clone https://github.com/casys-kaist/NeuPIMs.git
$ cd NeuPIMs
$ docker build . -t neupims
$ docker run -it -v .:/workspace/neupims-sim neupims
(docker) cd neupims-sim
(docker) git submodule update --recursive --init
(docker) ./build.sh

build docker image and installation

$ docker run -it -v .:/workspace/neupims-sim neupims
(docker) cd neupims-sim
(docker) ./brun.sh

run docker image

method 2 (Mannual)

Installation

$ git clone https://github.com/casys-kaist/NeuPIMs.git
$ cd NeuPIMs
$ git submodule update --recursive --init

Build

$ mkdir build && cd build
$ conan install .. --build missing
$ cmake ..
$ make -j

Run Simulator

$ cd ..
$ ./brun.sh

Baselines

  1. NPU-only: Codes on npu-only branch, all operations in LLM batched inference are executed on NPU.
  2. NPU+PIM: Codes on npu+pim branch, attention GEMV operations on PIM. PIM is single row buffered PIM on this baseline.
  3. NeuPIMs: Codes on main branch, we use dual row buffer PIM and sub-batch interleaving technique. (Sub-batch interleaving is enabled only when batch size>=256)

Citation

If you use NeuPIMs for your research, please cite our paper:

@inproceedings{10.1145/3620666.3651380,
author = {Heo, Guseul and Lee, Sangyeop and Cho, Jaehong and Choi, Hyunmin and Lee, Sanghyeon and Ham, Hyungkyu and Kim, Gwangsun and Mahajan, Divya and Park, Jongse},
title = {NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing},
year = {2024},
doi = {10.1145/3620666.3651380},
booktitle = {Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3},
series = {ASPLOS '24}
}

neupims's People

Contributors

hbbr0312 avatar

Stargazers

XinYao avatar Xilang Zhou avatar Yan Wang avatar  avatar Yiwei Yang avatar Jihoon Jang avatar  avatar Seong Hoon Seo avatar HJK avatar Wonhyuk Yang avatar Tongxin Xie avatar leliyliu avatar  avatar Shixin Zhao avatar Soojin Hwang avatar Boyu Tian avatar webber avatar Haomin Li avatar  avatar Minseok Seo avatar Yue Pan avatar Xiurui Pan avatar Hongsun_Jang avatar

Watchers

Youngjin Kwon avatar

neupims's Issues

Typo in model config file

I think this file has model parameters of gpt3-30B and not 175B. Could you please check if this is a typo?
neupims_configs/model_configs/gpt3-175B.json

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.