From the README, you are actually giving a rationale for GPU compute generally when explaining the choice of CUDA:

Note that we are primarily targeting CUDA - the performance, cost, and power usage of cheap GPUs like the Tesla P4 and GTX 1060 is too good to ignore. We'll make our best effort to support CPU wherever possible for current and future functionality but our emphasis is on performant latency-sensitive tasks even with low-end GPUs like the GTX 1060/Tesla P4 (as of this writing roughly $100 USD on the used market - and plenty of stock!).

I understand that nVidia dominates the GPU market, and that CUDA is ubiquitous, but using CUDA puts your project in the ironic position of effectively requiring closed-source software on users' machines for usable performance, which would seem to be antithetical to Willow's stated ethos (and additionally asks users to give money to a company that's notoriously hostile to OSS).

I'm not really hugely familiar with the GPU compute landscape, but maybe a first step could be for the project to use HIPIFY to (partially?) automate the generation of HIP code, which would at least provide a performant alternative without requiring closed-source drivers and on hardware from a far more open-source friendly company - and AIUI HIP is supported on the open-source amdgpu driver.

I don't know what the picture is like for Intel GPU hardware, but finding a solution there would be great too, since their integrated graphics are ubiquitous and would presumably still be better than CPU-only?

There's also OpenCL of course but I don't know what the features/porting situation is like there.

Related to #6:

https://opennmt.net/CTranslate2/python/ctranslate2.models.Whisper.html

SpeechT5 is included because it's in Transformers and it's an easy first pick for TTS.

There are several others (in no particular order):

Tortoise

Coqui

Toucan

MMS

Now that WIS has been released I'm very interested in feedback from the community to evaluate different engines, voices, etc so we can select the best default for future versions of WIS.

There's no licence information I can find about this project overall, apart from it being described as "open source" in the about section. What is the overall licence, and can you add a licence file and licence headers to the source files please? Thanks!

As the title says, I ran the utils.sh run command in the background, which successfully created the docker containers and had the app running. Given that the docker compose file had the container set to restart unless stopped, I figured if I rebooted the system as a test the containers would come back online on their own, but as it turns out only the WIS container came back, and the nginx one was no longer present in the container list. Seems as a result the WIS container was hung trying to restart.

I'll admit I'm not the most experienced docker user 😆 so I may be misunderstanding what is in the docker compose file. I did change the restart policy on both containers to "always" instead of "unless-stopped" and that made it so when I reboot they both come back online on their own without me having to log in and manually run the utils.sh. Just figured I'd call it out in case that is unintended, if it is intentional then please disregard 😄

EDIT: Figure this is understood, but this is on the wisng branch not the main branch.

We currently use libav python bindings for the handling of Opus frames -> WAV for Whisper. This is currently done via a file on disk, which is gross for a variety of reasons.

As a first pass we should at least try BytesIO or something.

For cards with insufficient memory we should probably have some basic heuristics to skip loading models that don't fit or whatever, possibly by user configured priority. Just skipping TTS would allow all Whisper models on 3GB cards, which should really be the minimum (GTX 1060 3GB).

As the title says. I was able to use detect_language true, but I would like to force a specific language - as usage now is a bit of a hit and miss. Of course it might be planned, i just would like it to happen wit a bit of prio hence the ticket

Currently for WebRTC ASR we do the following:

• Client opens page
• Media stream is started and recorder runs on server side
• User clicks stop and the recorded file is sent to Whisper
• Whisper sends ASR results back to client over data channel

Ideally what we would do is:

• Client opens page
• Media stream is started (for now)
• User clicks start to start recording
• User clicks stop and the recorded file is sent to Whisper
• Whisper sends ASR results back to client over data channel
• User can repeat start/stop as often as they would like

In this approach the user has better control of when the recording actually starts. This will work well when integrated with the speech microphone because we can trigger start/stop over the datachannel with record button press and release (push to talk, essentially). Not only that, they can record multiple statements in the same WebRTC session.

FastAPI is self-documenting as long as you don't hack it up and manually return responses like us. We need to define request/response the "FastAPI way" so the self-generated docs actually match code.

Way down the road (and I'm not sure we're going to stick with this approach anyway):

https://opennmt.net/CTranslate2/parallel.html

Chunking currently eats VRAM (totally understood). However, given the willow short speech segments intended application we should allow the user to disable chunking and return cleanly if the duration is over the default 30s

When testing Infer PROD with other people, we sometimes receive transcriptions for the audio stream supplied by the other user. I do not fully grok the WebRTC code yet, but I am suspicious of the global variables here and here.

This variable needs cleaned up when stop is called https://github.com/toverainc/air-infer-api/blob/13f7b5c15c6a6e525f692549d3162572de935e4d/main.py#L534

Issue is obvious. If you just call stop two times it will rerun the whisper model using leaked memory.

I don't like the sound of my voice :)

Is it possible to publish a Docker image so we can just docker run --gpus=all toverainc/willow-inference-server and have it work? That would make it trivial to launch our own server.

I know Whisper works in 30 second chunks but as one example transformers (via "pipeline") supports this.

From testing with an MD on Sunday it's clear they're going to want to ramble on in cases so we need to support audio longer than 30 sec.

For inspiration here's a project that according to the ctranslate2 docs supports this:

https://github.com/guillaumekln/faster-whisper

I think aiortc is the best approach from a latency standpoint - the media/audio is in memory until the moment Whisper is triggered and the audio is passed to it.

However, I'm skeptical that anything implemented in Python can handle large amounts of traffic. We're currently using a packetization interval of 20ms which translates to at least 50pps (packets per second) per direction per media stream. I've seen this present a challenge with the most optimized C/C++ implementations for media handling. We also potentially have the option of adjusting the ptime parameters of Opus because two-way latency isn't an issue for us and Opus implements forward error correction and packet loss concealment so losing a packet representing a larger chunk of audio with a higher ptime isn't as much of a concern as it is with other codec. We will also need to test is aiortc supports FEC and PLC...

Larger ptimes also drastically reduce the amount of bandwidth consumed.

Nothing that can't be done by throwing hardware at it and we'll probably have GPU and other procesing issues first anyway... Maybe?

When running any docker command like in ./build.sh I needed to add sudo to make it work, e.g.:

DOCKER_BUILDKIT=1 sudo docker build -t willow-inference-server:"$TAG" .

TLS certificates kind of complicate the setup a bit. It would be good if the WIS either exposed both HTTPS and HTTP ports (so I could ignore the TLS port for an internal setup), or it just didn't use TLS at all, so I can run my own ingress in front.

I think that running TLS termination inside the container is too much coupling, as most users would either be running their own TLS, if they want their stuff accessible from the outside, be running a VPN, making TLS redundant, or just not be exposing the server to the internet.

There are likely a few other places we can shave MS from - low priority but we'll track here.

Now that we're actually using requirements.txt we should prune it to make sure we really need all of that (currently at 118 deps).

The docker build is pretty messy right now. We're probably using the wrong base image (pytorch - and we install a new version anyway) and it doesn't support optional potentially more perf enhancing libs supported by ctranslate2:

https://github.com/OpenNMT/CTranslate2/blob/master/docker/Dockerfile

Is it worth updating our python runtime to 3.11 (for perf)?

Some users want TTS. Some want ASR. Some want both. Some want different models (large, medium, all, etc). Don't load functionality, models, etc the user doesn't want/need.

Tweak API responses for expected routes accordingly.

We should have a route to return stats of the system:

• HW Info (GPU name, mem, etc)
• Current GPU CPU
• Current GPU Mem
• Possibly others

Torch can do most of this (we already get everything other than utilization):

https://pytorch.org/docs/stable/generated/torch.cuda.utilization.html

I've heard rumors of someone creating an implementation that can create embeddings from user provided voice input. If we could have an API endpoint (preferably in conjunction with nice UI) to allow users to do this I think that would be a very cool feature :).

Hi there,

I guess my first comment is that GitHub discussions aren't enabled on this repo, so I'm putting this here as an issue 🙃.

I'm going to run this on a Proxmox server, and because I want to share the GPU across multiple services, a VM won't work well. Instead I would like to use an LXC.

Problem is, docker doesn't play nice with LXCs. So if there is a way to do a "bare metal" install (or one specifically designed for an LXC if you have a lot of Proxmox users) instead of docker, it should help a lot for those running other services such as Frigate NVR, Plex/Emby/Jellyfin, Stable Diffusion, Compreface, etc since LXC can share a GPU.

I hope this is a possibility, but I'll confess I'm not advanced enough to know how to make it possible.

https://docs.gunicorn.org/en/stable/settings.html#timeout

We want to enable gunicorn process management of workers. We need a value short enough to be useful but long enough to allow for worker startup across a wide variety of environments/CPUs/GPUs/etc.

Need to make a docker-compose for this project with Traefik to get a legit certificate from LE

Something happened that has drastically increased the WebRTC connection times... Still works and connects eventually but it's currently painfully slow.

For the WebRTC dynamic media port range the default docker daemon settings will launch $NUMPORTS of userspace docker UDP proxies - which takes a long time and is a waste of resources. We should dramatically warn when they don't have /etc/docker/daemon.json:

{
    "userland-proxy": false,
    "iptables": true,
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

We may be able to get this via the Docker API but needs more research.

We should also update the default port range to ~10 ports or something so even if they miss the warning, don't know how to do it, or don't need more concurrent clients even default userspace UDP proxies aren't a big deal.

For various clients like the emerging electron client, Home Assistant, web clients, etc we should have a known/fixed discovery type endpoint clients can query to discover the capabilities, models, etc of the server instance.

We should figure out a generic way to do and flag python profiling everywhere for performance regression testing

When whisper doesn't get speech it tends to return "you" as the ASR infer output. We should check for this condition and other variants and return something more meaningful to the user.

Response times are already fantastic but I'm wondering if there's a way to provide output progress during model execution earlier via the datachannel.

Looking through the ctranslate2 docs it doesn't look like it's possible but keeping this issue here as a placeholder.

the title says it all having WIS as home assistant addon will lower the barrier to entry for many people.

This project is amazing keep up the amazing work !!

I tried testing the TTS using a generated text response from my HA instance as follows:
Currently, the weather is 55 degrees with partly cloudy skies. Under present weather conditions, the temperature feels like 55 degrees. In the next few hours you can expect more of the same, with a temperature of 55 degrees.

What I noticed is the TTS generated silence for the number 55, but spoke all the other text. Seems it does not know how to handle numeric values?

I also noticed it did similar when trying to report time, such as 8:55AM. I also didn't yet try it, but I imagine it may have similar trouble handling date strings as well. Maybe there's a way to have it handle these specific numeric formats?

EDIT: just tried the string "Today is Thursday, June 01 2023." and it was silent on all the numbers. Also tried "Today is Thursday, June 1st 2023." and it says "st" on the 1st part, silence on all other numbers.

The latest release of ctranslate2 includes a parameter to fetch and convert a specific revision of the model from HF:

https://github.com/OpenNMT/CTranslate2/releases/tag/v3.11.0

We should implement this with the model download scripts for whisper via ctranslate and speecht5.

Hello,

As I'm trying to build on an arch linux based installation with docker up and running, I was faced with the following error message:

ERROR: BuildKit is enabled but the buildx component is missing or broken.
       Install the buildx component to build images with BuildKit:
       https://docs.docker.com/go/buildx/

Installing the docker-buildx package fixed it, but it might be worth mentioning it in the README.

On the client side (Electron/browser) our WebHID lib supports various dictation microphones.

When one of these devices is available as an audio source we should enforce/select usage of it for audio input.

Because we initialize the lib early and know if we have a supported device we can probably use that supported device type to select the audio source device (or even just regex match SpeechMike or PowerMic).

Later this week I will be receiving my test configuration for CPU only mode. I will be addressing:

• More conditionals for run.sh and others for systems that don't have GPUs
• Performance improvements on CPU only platforms
• Probably a few other random things!

Track on:
Max bandwidth - 70 kbps
Max pps - 59 pps

Track off:
Max bandwidth - 5 kbps
Max pps - 5

Overall, excellent!

Hello,
Managed to spin up an instance of WIS, and everything appears to be starting up correctly. However when I try to access the RTC page, it seems to try to make a connection but does not get any further. Here is the log data I can see:
iceConnectionLog disconnected signalingLog complete iceConnectionLog checking localDescription offer { "type": "offer", "sdp": "v=0\r\no=- 2102021303406714890 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS\r\nm=audio 64444 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126\r\nc=IN IP4 redacted\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=candidate:2850786284 1 udp 2122260223 192.168.17.1 64442 typ host generation 0 network-id 1\r\na=candidate:775075476 1 udp 2122194687 192.168.84.1 64443 typ host generation 0 network-id 3\r\na=candidate:3114883702 1 udp 2122129151 192.168.1.73 64444 typ host generation 0 network-id 2 network-cost 10\r\na=candidate:1655339575 1 udp 1685921535 redacted 64444 typ srflx raddr 192.168.1.73 rport 64444 generation 0 network-id 2 network-cost 10\r\na=candidate:3609487732 1 tcp 1518280447 192.168.17.1 9 typ host tcptype active generation 0 network-id 1\r\na=candidate:1358779404 1 tcp 1518214911 192.168.84.1 9 typ host tcptype active generation 0 network-id 3\r\na=candidate:3345397998 1 tcp 1518149375 192.168.1.73 9 typ host tcptype active generation 0 network-id 2 network-cost 10\r\na=ice-ufrag:3Tc7\r\na=ice-pwd:wcTlKOSBA2XDTkqkNxd/7hJv\r\na=ice-options:trickle\r\na=fingerprint:sha-256 45:DB:A6:1A:5D:72:A9:76:1A:F4:66:84:67:F2:39:A8:F4:D8:0A:C9:98:E1:A2:6F:C5:6B:FC:D2:B2:F8:16:1D\r\na=setup:actpass\r\na=mid:0\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=sendrecv\r\na=msid:- 568268e5-89b2-4660-a7b5-2b7de2060848\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=rtcp-fb:111 transport-cc\r\na=fmtp:111 minptime=10;useinbandfec=1\r\na=rtpmap:63 red/48000/2\r\na=fmtp:63 111/111\r\na=rtpmap:9 G722/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:13 CN/8000\r\na=rtpmap:110 telephone-event/48000\r\na=rtpmap:126 telephone-event/8000\r\na=ssrc:3526239653 cname:Kwlyo7DLJDeRJm5R\r\na=ssrc:3526239653 msid:- 568268e5-89b2-4660-a7b5-2b7de2060848\r\nm=application 64447 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 redacted\r\na=candidate:2850786284 1 udp 2122260223 192.168.17.1 64445 typ host generation 0 network-id 1\r\na=candidate:775075476 1 udp 2122194687 192.168.84.1 64446 typ host generation 0 network-id 3\r\na=candidate:3114883702 1 udp 2122129151 192.168.1.73 64447 typ host generation 0 network-id 2 network-cost 10\r\na=candidate:1655339575 1 udp 1685921535 redacted 64447 typ srflx raddr 192.168.1.73 rport 64447 generation 0 network-id 2 network-cost 10\r\na=candidate:3609487732 1 tcp 1518280447 192.168.17.1 9 typ host tcptype active generation 0 network-id 1\r\na=candidate:1358779404 1 tcp 1518214911 192.168.84.1 9 typ host tcptype active generation 0 network-id 3\r\na=candidate:3345397998 1 tcp 1518149375 192.168.1.73 9 typ host tcptype active generation 0 network-id 2 network-cost 10\r\na=ice-ufrag:3Tc7\r\na=ice-pwd:wcTlKOSBA2XDTkqkNxd/7hJv\r\na=ice-options:trickle\r\na=fingerprint:sha-256 45:DB:A6:1A:5D:72:A9:76:1A:F4:66:84:67:F2:39:A8:F4:D8:0A:C9:98:E1:A2:6F:C5:6B:FC:D2:B2:F8:16:1D\r\na=setup:actpass\r\na=mid:1\r\na=sctp-port:5000\r\na=max-message-size:262144\r\n" } iceGatheringLog complete iceGatheringLog gathering signalingLog new added track to peer connection

When I open the debug console, I see this response from asr:
{ "sdp": "v=0\r\no=- 3893853050 3893853050 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1\r\na=msid-semantic:WMS *\r\nm=audio 10035 UDP/TLS/RTP/SAVPF 111\r\nc=IN IP4 172.17.0.2\r\na=recvonly\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=mid:0\r\na=msid:36c53f4f-8f1f-458c-a189-9c1747570917 33ffaf28-0aeb-4bec-9886-fdb0d7e452c3\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=rtcp-mux\r\na=ssrc:2402540680 cname:5246c902-d951-4bbd-b3ab-2f4ade706e73\r\na=rtpmap:111 opus/48000/2\r\na=candidate:9333c84bcc1b0bf56713df9036e6b4d9 1 udp 2130706431 172.17.0.2 10035 typ host\r\na=candidate:c58f5770074e5a6227e87732712d9300 1 udp 1694498815 redacted 10035 typ srflx raddr 172.17.0.2 rport 10035\r\na=end-of-candidates\r\na=ice-ufrag:qcEp\r\na=ice-pwd:IQ2ymsMj77ZqsnuWS6As6h\r\na=fingerprint:sha-256 6F:B3:6F:97:62:8A:7B:01:8D:A9:4E:0D:D5:B4:D9:E0:B4:99:97:DF:85:53:BA:B5:AE:07:54:5C:1F:BB:4C:32\r\na=setup:active\r\nm=application 10035 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 172.17.0.2\r\na=mid:1\r\na=sctp-port:5000\r\na=max-message-size:65536\r\na=candidate:9333c84bcc1b0bf56713df9036e6b4d9 1 udp 2130706431 172.17.0.2 10035 typ host\r\na=candidate:c58f5770074e5a6227e87732712d9300 1 udp 1694498815 redacted 10035 typ srflx raddr 172.17.0.2 rport 10035\r\na=end-of-candidates\r\na=ice-ufrag:qcEp\r\na=ice-pwd:IQ2ymsMj77ZqsnuWS6As6h\r\na=fingerprint:sha-256 6F:B3:6F:97:62:8A:7B:01:8D:A9:4E:0D:D5:B4:D9:E0:B4:99:97:DF:85:53:BA:B5:AE:07:54:5C:1F:BB:4C:32\r\na=setup:active\r\n", "type": "answer" }

Let me know if any other info would be helpful here, thanks! :)

Given the nature of webrtc and the support for those clients the ability for AIA to support $NUM clients is dependent on having free dynamic media ports available. We need to make sure when a datachannel connection closes that we free it and the port it has allocated for the multiplexed UDP session.

When we have the infrastructure in place (images pushed to Docker Hub, etc) an inexperienced user should be able to run a single command and get up and running - possibly without even cloning the repo.

We hate it but possibly even the classic curl | bash one liners you see all over the internet...

https://github.com/THUDM/ChatGLM-6B

https://huggingface.co/THUDM/chatglm-6b-int4

It would be best if you could provide some guidance. Thank you.