infini-ai-lab / triforce Goto Github PK
View Code? Open in Web Editor NEW[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
Home Page: https://infini-ai-lab.github.io/TriForce/
triforce's People
Contributors
Stargazers
Watchers
Forkers
awaitfuture lihuibng strategist922 eltociear dyl777 sunningily jjhw apollohuang1 preminstrel junzhang-zjtriforce's Issues
Example code to run batched inference?
Nice work! In the paper I saw this batched result:
But examples like https://github.com/Infini-AI-Lab/TriForce/blob/main/test/on_chip.py only use batch size=1. Does the code supports batched speculative inference?
Questions about end2end time cost of the inference request
Hi, thanks for your great job for LLM decoding process. I tested the code and got the expected decoding speedup for llama2-7B, but it seems that the end2end time cost does not change too much? (61s -> 58s). I profile the inference process and it seems that the prefill process occupied the vast majority of inference time. Is the conclusion same with your experiments? Thanks !
Method: TriForce
Dataset: gs
Spec Args: {'budget': 4096, 'chunk_size': 8}
Draft: /mnt/bn/multimodel/models/official/llama-68m
Target: /mnt/bn/multimodel/models/official/NousResearch--Yarn-Llama-2-7b-128k/model
Prefill Length: 124928
Generation Length: 256
Gamma: 6
Sampling Method: top_k = -1, top_p = 0.9, temperature = 0.6
Log CSV: None
######################################################################################
[draft run] capturing graph for 0 (probs=True, temp=0.6, top_p=0.9)...
[draft run] capturing graph for 1 (probs=True, temp=0.6, top_p=0.9)...
[draft run] capturing graph for 2 (probs=True, temp=0.6, top_p=0.9)...
[draft run] capturing graph for 3 (probs=True, temp=0.6, top_p=0.9)...
[draft run] capturing graph for 4 (probs=True, temp=0.6, top_p=0.9)...
[draft run] capturing graph for 5 (probs=True, temp=0.6, top_p=0.9)...
[draft run] capturing graph for 6 (probs=True, temp=0.6, top_p=0.9)...
[draft run] capturing graph for 7 (probs=True, temp=0.6, top_p=0.9)...
[draft run] capturing graph for 8 (probs=True, temp=0.6, top_p=0.9)...
[model verify] capturing graph for spec len 6 (probs=True, temp=0.6, top_p=0.9)...
[Full Cache] Cached: 0 | Budget: 125200
[Retrieval Cache] Budget: 4096 | PreFill: 124928 | Chunk Size: 8 | Chunks: 15616 | Select Sets: 512
[StreamingLLM Cache] Start Size: 16 | Recent Size: 234 | Gamma: 6 | Real Budget: 259 | Cached: 0
tokenized_prompts length: 20
Autoregressive Warmup: 100%|████████████████████████████████████████████████| 1/1 [01:01<00:00, 61.31s/it]
Autoregressive Test: 100%|██████████████████████████████████████████████████| 1/1 [01:01<00:00, 61.73s/it]
[Autoregressive] average latency: 51.494828425347805 ms
TriForce Warmup: 100%|██████████████████████████████████████████████████████| 3/3 [02:53<00:00, 57.91s/it]
TriForce Test: 100%|██████████████████████████████████████████████████████| 20/20 [19:33<00:00, 58.66s/it]
average acceptance rate (NOT per token): 0.7204096470358102
[TriForce] average latency: 24.157854936546297 ms
[E2E Speedup]: 2.1315977167925535
Does Retrieval w/o Hierarchy test with spec decoding?
I have a question on paper results.
Does Retrieval w/o Hierarchy test with normal speculative decoding (Retrieval cache + Full cache)?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.