Comments (8)
Can you send me your demo-config.json?
PS: the original demo-config.json is just a demo. You need to modify the content to generate a tree you want.
from sequoia.
{
"acceptance_rate_vector": "acceptance-rate-vector.pt",
"max_depth": 15,
"max_budget": 128,
"draft_time": 0.0003,
"valid_budget": [1, 2, 4, 8, 16, 32, 64, 128],
"target_time":[0.025, 0.025, 0.025, 0.025, 0.025, 0.027, 0.030, 0.035],
"dst": "demo_tree.pt"
}
p = [0.0000, 0.4803, 0.1104, 0.0576, 0.0373, 0.0265, 0.0211, 0.0170, 0.0135,
0.0113, 0.0093, 0.0087, 0.0075, 0.0067, 0.0058, 0.0061, 0.0049]
might be a proper example to generate a tree of size of 32.
Some explanation: draft_time is the time for one draft model's forward pass. target_time is the time for one draft model's forward pass corresponding to the valid budget.
from sequoia.
Thanks for the fast reply and the explanation! The demo-config.json
is the same as that in the repo.
{
"acceptance_rate_vector": "acceptance-rate-vector.pt",
"max_depth": 10,
"max_budget": 128,
"draft_time": 0.38,
"valid_budget": [1, 2, 4, 8, 16, 32, 64],
"target_time":[10, 10, 10, 12, 14, 18, 27],
"dst": "demo_tree.pt"
}
The time above was assumed to be in the unit of ms
.
acceptance_rate_vector:
tensor([0.0000, 0.6342, 0.1079, 0.0570, 0.0225, 0.0195, 0.0150, 0.0045, 0.0030, 0.0120, 0.0045, 0.0075, 0.0045, 0.0060, 0.0030, 0.0015, 0.0030, 0.0015, 0.0030, 0.0000, 0.0030, 0.0030, 0.0030, 0.0000, 0.0015, 0.0000, 0.0015, 0.0000, 0.0000, 0.0000, 0.0015, 0.0015, 0.0015, 0.0735])
which is similar to the acceptance vector in the repo too.
I can first try your demo-config.json
and the acceptance_vec above too. It seems that our acceptance vec sizes are also different. About the times in the config, I previously assume that the unit in the numbers are not important; i.e. if we simultaneously scale the draft_time and the target_time by the same multiplicity, the resultant tree is invariant. I don't know if this is the correct assumption.
from sequoia.
Here is some updates. I tried your config.json
with my original acc_rate_vec.pt
, the generated tree becomes normal, of size 32.
From the ablation test, it looks like the target time [10, 10, 10, 12, 14, 18, 27]
(unit ms) I used is the key reason that caused the optimal tree to be of size 4. It looks like the algo is pretty sensitive to the target_time profile. In contrast to the above target_time profile, [ 11, 11, 11, 11, 11, 20, 31]
(unit ms) generate tree size 16. The two are not quite different, yet the resultant tree sizes vary a lot.
By the way, is it true that the generated tree sizes can only be numbers from "valid_budget": [1, 2, 4, 8, 16, 32, 64] ? And to allow for more tree size numbers, in the config.json, the data points of more valid_budgets have to be provided there?
from sequoia.
Yes, the generated tree sizes can only be numbers from "valid_budget".
from sequoia.
How to determine the optimal depth and budget? Can you share the config.json
that generated the example A100 and L40 growmaps in this repository?
from sequoia.
Some explanation: draft_time is the time for one draft model's forward pass. target_time is the time for one draft model's forward pass corresponding to the valid budget.
Is there a typo here? Shouldn't target_time
be the time for a verification pass?
from sequoia.
Some explanation: draft_time is the time for one draft model's forward pass. target_time is the time for one draft model's forward pass corresponding to the valid budget.
Is there a typo here? Shouldn't
target_time
be the time for a verification pass?
I have the same question.
"draft_time": 0.0003
What is the unit of this value? I don't think any model can complete the forward pass within 0.0003 seconds?
from sequoia.
Related Issues (12)
- Error `p.attn_bias_ptr is not correctly aligned` when testing HOT 1
- Tensor shape mismatch when computing apply_rotary_pos_emb HOT 5
- Integration with Lit-GPT HOT 2
- data loading timing and disk use
- Thanks for your good work.
- Is there any benchmark that compares Sequoia against vanilla speculative decoding? HOT 2
- The support on vLLM? HOT 1
- How to benchmark for speedup and acceptance rate? HOT 7
- Estimate the number of generated tokens per step from the acceptance-rate-vector? HOT 1
- Question on tree search algorithm HOT 3
- Work On CPU
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sequoia.