Comments (7)
Thanks @dacorvo we'll keep the ticket open in the interim, but will assume you have the next step
from transformers-neuronx.
Hi @dacorvo,
Could you also share the parameters you call the GPT2ForSampling.sample() and generate()?
The default behavior of sample and generate is significantly different:
*.sample() uses topk (k=50)
- to use same topk in generate, it would expect generate(topk=50)
It would be really helpful if you could share the minimum script including these commands, and the instance type you used to reproduce your profiling results.
from transformers-neuronx.
Hi @mrnikwaws, thank you for your answer.
I think top_k
is by default set to 50 in transformers. I set it explicitly anyway, but I got the same results.
Here is the link to my script: https://github.com/dacorvo/transformers-neuronx/blob/inference_tests/test_gpt2.py
from transformers-neuronx.
I did a few more tests on an 24xlarge instance, which has 6 devices / 12 cores.
Using a batch size of 16 and generating 1000 tokens, the test lasts long enough so that I can get meaningful numbers using neuron_top
in parallel.
What I see is that when using the GPT2ForSampling.sample()
method directly, the 12 cores are used up to 60 %, while when using the wrapper generate()
method, they are only used up to 20 % (virtual CPU usage looks equivalent).
This is purely speculative, because I have no idea of what actually happens behind the scene, but in a previous life developing software for hardware accelerators, this was often the sign of too much latency in the host process feeding the hardware.
from transformers-neuronx.
I investigated a bit further and the top-k sampling code is indeed much slower in transformers, which creates latency between the hardware invocations. I think I might have identified why the sample_loop()
is faster and I will see if I can align the transformers top-k implementation, and confirm this solves this issue.
from transformers-neuronx.
I have modified the transformers sampling loop in optimum-neuron
:
huggingface/optimum-neuron#130
The performances of the generation through transformers
generate()
are now equivalent to those using the transformers-neuronx
custom sampling loop.
FYI, I submitted the issue to the transformers
team but at this stage it is not going to be integrated in the core library.
Said differently, you will need to go through the optimum-neuron
package to get the optimized generation code: the change will not benefit to the transformers-neuronx
HuggingFaceGenerationModelAdapter
class.
from transformers-neuronx.
Thanks @dacorvo we're taking a look at what we can do to alert customers on this issue in our docs / through warning messages.
from transformers-neuronx.
Related Issues (20)
- How to use generate() with inputs_embeds HOT 2
- Mixtral config issue -- not handling null well HOT 8
- Generate Llama 2 from Embeddings HOT 5
- Infering logits from `model.forward` for the entire batch instead of the last forward's output. HOT 6
- Support for MPT model HOT 1
- `stopping_criteria_list(input_ids, probs)` does not check for the correct sequence. HOT 4
- User feedback when compiling and reloading a large model HOT 1
- Issue while compiling Mistral 7B 0.2 Instruct HOT 5
- Backward compatibility with saved llama 2 compiled artifacts HOT 1
- NaN outputs when masking llama model inputs HOT 8
- Improve Neuron model loading time HOT 4
- Add support for `gemma` models HOT 1
- Add support for Baichuan-13B model
- Latest changes introduced for continuous batching break Mixtral model HOT 5
- llava support HOT 3
- Any plan to support Qwen-2 Model
- Neuron model NEFFs are dependent on the python path HOT 2
- Not able to load llama 3 70b on inf2.24xlarge instance HOT 6
- Gibberish output for princeton-nlp/Sheared-LLaMA-1.3B with continuous batching HOT 2
- [Question] BasicTransformerBlock
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transformers-neuronx.