Are ONNX Models Necessary? about silero-vad HOT 15 CLOSED

I thought we can export ONNX model if use JIT script models even though got if statement?

Can you provide a minimal working export example with if?

Sure, https://colab.research.google.com/drive/1sl33VBqT8fy46zyrhU1gVjWJOKw0Ya1i?usp=sharing

Nice, many thanks
We discussed this issue with colleagues this morning
This link also helps - https://pytorch.org/docs/stable/onnx.html#tracing-vs-scripting

Tbh, I did not know that control flows are supported, but only in scripting model
We could replicate this on toy examples as well
But with real models we face a cryptic error we cannot yet solve

Probably we may opt for the following solution

vad.jit 8/16 kHz 
vad_q.jit 8/16 kHz 
vad.onnx 16 kHz only

from silero-vad.

I thought we can export ONNX model if use JIT script models even though got if statement?

Can you provide a minimal working export example with if?

Sure, https://colab.research.google.com/drive/1sl33VBqT8fy46zyrhU1gVjWJOKw0Ya1i?usp=sharing

from silero-vad.

I think it should be possible to load PyTorch models in most languages like e.g. with DJL from Amazon for Java.

Also, I am not sure, but isn't it possible to convert load pretrained PyTorch models and to convert them to ONNX models anytime? So if someone needs them he could do the conversion by himself. I remember that I read something like that, but not sure about the details or if there were requirements.

I have not used the ONNX models though. Maybe someone will respond that has used the ONNX models and could explain why it would be good if they would remain 😄

from silero-vad.

ONNX models are created from nn.Module or JIT models in torch.

The key issue for me here is not really maintaining or creating the models, but since we have been able to radically simplify everything, just leaving one VAD model for everything seems enticing (different SR, variable chunk size, much less code for iterators, etc).

The problem is that ONNX does not support if statements and new JIT models have one if statement inside.

from silero-vad.

I thought we can export ONNX model if use JIT script models even though got if statement?

I prefer ONNX model because don't want to install torch in my prod env

from silero-vad.

I thought we can export ONNX model if use JIT script models even though got if statement?

Can you provide a minimal working export example with if?

from silero-vad.

In case you guys don't want to share onnx anymore, can share jit model without quantization?
Currently I cannot export text enhancement model to onnx because of quantization operation

from silero-vad.

Currently I cannot export text enhancement model to onnx because of quantization operation

Text enhancement has a lot of logic under the hood, so no ONNX planned there by design
There is a lot of python inference code tucked in

from silero-vad.

Currently I cannot export text enhancement model to onnx because of quantization operation

Text enhancement has a lot of logic under the hood, so no ONNX planned there by design There is a lot of python inference code tucked in

Yeah, saw a lot of function in it like enhance text, unitoken_into_token, process_unicode etc
I can move it into my inference code, the only issue at exporting the te_model because of quantization operation

from silero-vad.

I thought we can export ONNX model if use JIT script models even though got if statement?

Can you provide a minimal working export example with if?

Sure, https://colab.research.google.com/drive/1sl33VBqT8fy46zyrhU1gVjWJOKw0Ya1i?usp=sharing

Nice, many thanks We discussed this issue with colleagues this morning This link also helps - https://pytorch.org/docs/stable/onnx.html#tracing-vs-scripting

Tbh, I did not know that control flows are supported, but only in scripting model We could replicate this on toy examples as well But with real models we face a cryptic error we cannot yet solve

Probably we may opt for the following solution

vad.jit 8/16 kHz 
vad_q.jit 8/16 kHz 
vad.onnx 16 kHz only

cool, later when you guys share vad.jit I might try export it to onnx
If I can :p

from silero-vad.

Yeah, saw a lot of function in it like enhance text, unitoken_into_token, process_unicode etc

I can move it into my inference code, the only issue at exporting the te_model because of quantization operation

Tucking this in was a design decision, because we wanted to package and support as little as possible
Also a TE model is a more niche model, so we just decided not to bother with publicly sharing those

from silero-vad.

Yeah, saw a lot of function in it like enhance text, unitoken_into_token, process_unicode etc

I can move it into my inference code, the only issue at exporting the te_model because of quantization operation

Tucking this in was a design decision, because we wanted to package and support as little as possible Also a TE model is a more niche model, so we just decided not to bother with publicly sharing those

Ohh ok got it
I guess my alternative choice is to train it myself like what you did.
Thanks

from silero-vad.

cool, later when you guys share vad.jit I might try export it to onnx

If I can :p

There are some low level cryptic errors for some reason, basically saying that some data type becomes int64 and fails in ONNX. But we have some custom modules, that we also intentionally tucked in. So if we cannot fix it, we will just say that ONNX works only for 16k.

We want to radically simplify everything and having more than one variety model is a no-no for future. We had to be inventive to have the best quality, you will see in the PR releases after the VAD release.

from silero-vad.

cool, later when you guys share vad.jit I might try export it to onnx

If I can :p

There are some low level cryptic errors for some reason, basically saying that some data type becomes int64 and fails in ONNX. But we have some custom modules, that we also intentionally tucked in. So if we cannot fix it, we will just say that ONNX works only for 16k.

We want to radically simplify everything and having more than one variety model is a no-no for future. We had to be inventive to have the best quality, you will see in the PR releases after the VAD release.

Ohhh, might be bug in onnx graph code
Ok cannot wait for the release :)

from silero-vad.

In the end we could solve all of the ONNX problems that we could identify (with our custom modules), but ended up with cryptic export errors nonetheless.
We will return to this in future, when we have more time / focus.

Turns out for "if-less" models we just used tracing, which produced horribly looking, but working models.

For models with ifs (they need scripting) we could beat the ONNX limitations and polish our custom modules to be compatible, but ended up with more cryptic errors.

We will have a final module by module debugging session some time 1Q next year.

For now we decided to opt for simplicity and collect feedback.

from silero-vad.