A converter and basic tester for rwkv V5 onnx
- fp16, fp32
- onnx opsversions 15/17/18
- file sizes > 2GB
- cuda, cpu, tensorRT, mps
Install with
pip install protobuf==3.20.0
A converter and basic tester for rwkv onnx
I locate the bug:
When I converted the 3B model, the size of graph_def was larger than 3GB. Onnx couldn't deal with files larger than 2GB. Could we output RWKV layer by layer?
i used the convert.py to convert the rwkv-5 model ,when i used the test.py to test if it was successful,i failed.
[?] Select model:
> RWKV_24_2048_32_15.onnx
['AzureExecutionProvider', 'CPUExecutionProvider']
[?] Select execution providers(use space bar to select checkboxes):
> [ ] AzureExecutionProvider
[ ] CPUExecutionProvider
[]
2024-05-18 15:30:52.673417593 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'PreTrainedTensor_4'. It is not used by any node and should be removed from the model.
2024-05-18 15:30:52.673432693 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'PreTrainedTensor_1'. It is not used by any node and should be removed from the model.
2024-05-18 15:30:52.673435491 [W:onnxruntime:, graph.cc:3593 CleanUnusedInitializersAndNodeArgs] Removing initializer 'PreTrainedTensor_0'. It is not used by any node and should be removed from the model.
tensor(float) 2048 24
(48, 2048)
(24, 40, 64, 64)
/.local/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
0%| | 0/19 [00:00<?, ?it/s]
Traceback (most recent call last):
File "rwkv-onnx/ds/../test.py", line 120, in <module>
logits, state, state2 = model.forward(token,state, state2)
File "rwkv-onnx/ds/../test.py", line 66, in forward
outputs = sess.run(output_names, inputs)
File "/.conda/envs/py10/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 220, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: instatewkv9 for the following indices
index: 0 Got: 40 Expected: 32
Please fix either the inputs/outputs or the model.
I have tried to convert the weight file which i download from https://huggingface.co/RWKV/rwkv-4-world-169m with convert.py
. I found the following problems.
convert.py
..att.
in convert.py
should be replaced with .attention.
.rwkv.blocks.0.attention.time_decay
is a 1d tensor. The headsnume
and headsize
in convert.py
can't unpacke from w[f"rwkv.blocks.0.attention.time_decay"].shape
I don't know how to fix the third problem.
The input state/out state is separated, I add some ops in rwkv-cpp-server to concat and reshape the state tensors so that every rwkv onnx model has two input tensors and two output tensors.
I use convert.py to convert the rwkv model: https://huggingface.co/BlinkDL/rwkv-4-world/blob/main/RWKV-4-World-0.1B-v1-20230520-ctx4096.pth
Convert to onnx model (FP32), run the model, the output data is NaN, the output name is name: matvec_1122_out
I added support for world models, but currently cannot run rwkv-onnx converted world models.
Is RWKV official interested in adding an ONNX implementation for RWKV6?😀
Here may be some useful references:
https://github.com/yuunnn-w/RWKV_Pytorch
Is the code able to quantize the model to int8?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.