Feature request: Speech2Text (Whisper) as chat input about chatbot-ui HOT 13 CLOSED

Good call. I think with that type of UX it would be great. I plan on making token usage visible, and adding Whisper hours to that would be pretty easy. I'll work on this - really like the idea.

from chatbot-ui.

Web speech API is a good option since it seems to be free. It's pretty powerful for what it is but
a) only works on Chrome and Firefox (not Brave) and
b) especially if you use out of distribution, slang, domain language or technical terms web speech api is inaccurate

See for yourself here [Chrome only].

Could be good as first implementation or option for the user.

In terms of UI for the feature, IMO Voice Control for ChatGPT extension has done the best job. I downloaded Chrome just to be able to use it. UX and Keyboard commands and ergonomics make a lot of sense.

from chatbot-ui.

One option could be to allow users to directly run the Whisper Model directly on their laptops, it would be for power users only, likely

from chatbot-ui.

I really like this idea. The only issue I see with it is that Whisper usage costs might add up pretty quickly for the user. I'll add this to my list and look into it some more. I've worked with the Whisper API so I could get it done quite quickly.

from chatbot-ui.

I could add it as an option. I think @markusstrasser is advocating for Whisper due to its performance. Extremely accurate.

from chatbot-ui.

Sounds great, looking forward to this feature!

from chatbot-ui.

I could add it as an option. I think @markusstrasser is advocating for Whisper due to its performance. Extremely accurate.

This extension might be helpful: C-Nedelcu/talk-to-chatgpt#45

from chatbot-ui.

Good point. At 0.006$/minute transcribed I estimate it to be around 3-15 cents per hour of active chatbot usage, depending on the type of convo. Maybe we can display 'minutes transcribed' and 'costs', possibly set a maximum budget per session/day etc. to clip the downside.

Would be great since it improves user's speaking ability as a freebie (whereas typing won't improve any further)

from chatbot-ui.

How about adding web speech api?

from chatbot-ui.

Thank you for making this! I was working on something similar but this blows it out of the water.

Would creating a button that sends an audiostream to a local instance of https://github.com/ggerganov/whisper.cpp be very difficult? I have a version of that running but I am terrible at TypeScript

from chatbot-ui.

Whisper.cpp has also a WebAssembly version that works in the browser alone https://github.com/ggerganov/whisper.cpp/tree/master/examples/whisper.wasm

from chatbot-ui.

I don't mind running whisper locally, not too much to setup, could we do a config setting for to and from speech endpoints, i would probably be looking at running https://github.com/askrella/speech-rest-api or something similar
I'd be happy to collaborate on a PR if this, or even an option for openapi whisper as well, I'm not sure how often I choose to use speech, but probably less if i'm weary of api costs

from chatbot-ui.

is this integrated? in the hosted version I do not see any microphone button for doing STT?

from chatbot-ui.