c-frame / sponsorship Goto Github PK

Finish text wrapping tests and PR to threejs repo

• AdaRoseCannon/aframe-htmlmesh#11

• networked-aframe/networked-aframe#425

• networked-aframe/networked-aframe#299

• networked-aframe/networked-aframe#418

• aframevr/aframe-inspector#649

Create a networked-aframe adapter for LiveKit open source SFU, it's based on the pion WebRTC stack (Go language).

The interesting part of this stack is using egress plugin to record the audio on the server and transcribe it with whisper for example, see recent experiments:

• https://twitter.com/livekitted/status/1648352311444652033 and repo https://github.com/atyenoria/livekit-whisper-transcribe
• https://twitter.com/livekitted/status/1646194790151909376
• https://twitter.com/grvydev/status/1663710002216075265 and repo https://github.com/GRVYDEV/S.A.T.U.R.D.A.Y

I'll need to record audio for a 3d meeting project for legal reasons, so I'll work on it.
My monthly sponsors could have access to it once I developed it. Please show your interest on this issue by adding a thumbs up and also being a monthly sponsor.
If this issue have enough interest, I'll write a proper documentation to self host LiveKit and using the adapter for networked-aframe.

• networked-aframe/networked-aframe#423

Create an aframe component for three-particle-fire

I actually already developed this component but currently in a private repo in a big project.
The work is mainly to open source it, create a new repository, add a README and example.

• aframevr/aframe-inspector#694

• aframevr/aframe-inspector#690

• aframevr/aframe-inspector#698

• networked-aframe/networked-aframe#422

This is a modified version of the uv-scroll.js hubs code that can be used with the aframe material component. Changes required were to use an aframe system and waiting for material to be loaded via the materialtextureloaded event.
Licensed under MPL 2.0

Usage:

• https://hubs.mozilla.com/labs/animating-textures-with-the-uv-scroll-component-pt1/
• https://hubs.mozilla.com/labs/animating-textures-using-the-uv-scroll-component-part-2/ and tweet
• https://hubs.mozilla.com/labs/animating-textures-using-the-uv-scroll-component-part-3/ and tweet

Finish integrating mozilla's hackweek-avatar-maker in a naf experience without needing to export the avatar, just transferring the config to the other participants and the glb is generated and optimized on the fly from the glb parts upon receiving the configuration.

Current working branch is
https://github.com/vincentfretin/hackweek-avatar-maker/tree/remove-style

that includes the following PRs:

• mozilla/hackweek-avatar-maker#37
• mozilla/hackweek-avatar-maker#38
• mozilla/hackweek-avatar-maker#23

That development would be on https://allocola.com behind a flag to be able to create those avatars instead of selecting the current default realistic avatars.

Generalize hover state to more elements and create a PR to threejs repo

• AdaRoseCannon/aframe-htmlmesh#9

This is the simple-water and SimpleWaterMesh hubs code that would be extracted in a separate repo to use in aframe projects.
Licensed under MPL 2.0

Changes required is removing typescript annotations, removing APP.store.state.preferences.materialQualitySetting usage and replacing HubsTextureLoader by TextureLoader.
I'm not currently using it in an experience but that would be a task to do if we want compatibility with hubs scene build with hubs-blender-exporter in aframe projects.

One of the contribution on it was merged there Hubs-Foundation/hubs#5510

networked-audio-analyser and local-audio-analyser hubs components (see audio-feedback.js) working with networked-aframe projects to know when someone is talking.
The local-audio-analyser component is using a local-media-stream-created event that need to be emitted when you select a mic.
Calculating a volume is not done if player-info component has a muted property set to true.
Add an optimization similar to the frame-scheduler system to calculate only one participant volume per frame instead of all participants volume in a frame. (That one I didn't implement yet)

Licensed under MPL 2.0

• aframevr/aframe-inspector#697

I'm currently using the speechSynthesis api for text to speech, but this api doesn't work in VR on Meta browser. Also the voice is different from one platform to another, using a male voice on a female avatar is funny but not for a customer :-)
The api is a bit tricky with the voices list that is async, you can read more on this article (7 dec. 2021, so some information may not be accurate anymore)

I'm working on a coqui cpu integration with the official docker image, integrating it to my existing server without GPU.
The "tts_models/multilingual/multi-dataset/your_tts" model (article) is actually quite good for English and French (That's funny for French that you have a enough good result with speaker_id="male-pt-3\n" and language_id="fr-fr")

The backend part will consist of a docker-compose file and one or several docker containers to generate the audio from text
suitable for a production usage (several users communicating with a gpt-3.5 agent at the same time in different rooms):

• nginx+gunicorn for WSGI (coqui-tts is a Python app)
• Probably using the proxy cache varnish to queue similar requests for the same text, generate once the wav, cache a few minutes the wav in memory to reply to all requests.

I'm working also on an aframe component that split the text on punctuation into chunks, does the fetch call for each chunk to the coqui tts service and play the audio chunk sequentially. For the fetch call and playing the audio file received, see their code

I'm working on it for my current project. When I'm done implementing it, I'll open source it in a private repo with instructions how to self host it and use the aframe component for my $10 tier monthly sponsors. The access to the repo will be public 4 months later.

Resources:

• Fabien experiments with coqui: https://twitter.com/utopiah/status/1651646682549964802 and https://twitter.com/utopiah/status/1651968536640667654
• Alternative coqui tts docker image https://github.com/fquirin/docker-containers/tree/main/SEPIA/Coqui-TTS when sepia implemented the MaryTTS API for use with HomeAssistant. The MaryTTS API changes are now upstream, but you can look at the Dockerfile and scripts there.

Alternatives:

• https://github.com/webaverse/tiktalknet using a server with an NVIDIA GPU, there is a Dockerfile is the repo, found via https://discord.com/channels/758943204715659264/977340924780748901/1076424376355917835
• https://www.inworld.ai (Free tier available) apparently used in the Niantic / 8thWall's Wol mixed reality experiment, see article

Create a separate repo in c-frame organization to host simple-navmesh-constraint component and publish it to npm.
See AdaRoseCannon/aframe-xr-boilerplate#13 (comment)

example:
https://glitch.com/edit/#!/aframe-ada-simple-navmesh-constraint---1-4-1
doc:
https://github.com/AdaRoseCannon/aframe-xr-boilerplate#simple-navmesh-constraintjs

• aframevr/aframe-inspector#688
• aframevr/aframe-inspector#689

• networked-aframe/networked-aframe#374