Comments (3)
Hi @DC2009. Unfortunately we don't currently have this as an option for hosts through the exposed feature API. Part of this is because it opens up room for error and confusion since you need to create both speech audio and speechmarks, we didn't want people to assume that you could pass in any audio file and get working lipsync. We also keep track of Polly usage so we can determine how much use people are getting out of our open source hosts solution. But you could certainly fork the repository and create a custom build that allows for this. There is only a single method you would need to override: AbstractTextToSpeechFeature._updateSpeech. At the bottom where we wait for the speechmarks and speech audio to be synthesized you could just skip that and use pre-existing objects.
You also do have full control over the visemes. You can manually blend them on and off using the AnimationFeature.setAnimationBlendWeight method.
from amazon-sumerian-hosts.
Instead of forking the repository, we decided to extend the class HOST.aws.TextToSpeechFeature and override a few methods in order to make it work without direct connection to Polly. I called this feature VcaFeature (from the name of our service).
I receive from our server a base64 audio and a string with speech marks, generated by Polly. The audio is played correctly using Babylon.
There is no lipsync. From what I found, LipsyncFeature listens to TextToSpeechFeature events, so I changed the name of our feature to TextToSpeechFeature. It seems LipsyncFeature listens properly to EVENT.play but I get the following error:
Uncaught (in promise) Error: Cannot interpolate property blendValueX to value NaN. Target value must be numeric.
at Function.interpolateProperty (AnimationUtils.js?1148:96)
at Blend2dState.setBlendWeight (Blend2dState.js?8163:105)
at AnimationLayer.setAnimationBlendWeight (AnimationLayer.js?beec:272)
at AnimationFeature.setAnimationBlendWeight (AnimationFeature.js?ebde:706)
at eval (PointOfInterestFeature.js?dee5:869)
at Array.forEach (<anonymous>)
at PointOfInterestFeature.update (PointOfInterestFeature.js?dee5:803)
at eval (HostObject.js?75e7:84)
at Array.forEach (<anonymous>)
at HostObject.update (HostObject.js?75e7:83)
at r.callback (host.js:26)
at e.notifyObservers (babylon.js:16)
at t.render (babylon.js:16)
at t._renderFrame (babylon.js:16)
at t._renderLoop (babylon.js:16)
during the host update
23 | // Add the host to the render loop
24 | const host = new HOST.HostObject({ owner: character });
25 | scene.onBeforeAnimationsObservable.add(() => {
> 26 | host.update();
| ^ 27 | });
28 |
Any idea what is going on? Is it possible to extend the textToSpeechFeature class like that or is it better to fork the project? We didn't want to for the project as it's quite new and possibly will change.
from amazon-sumerian-hosts.
Hi @DC2009. I think you're on the right track for extending the TextToSpeechFeature class as opposed to forking, that's definitely a valid alternative. The error you pasted above is actually related to the PointOfInterestFeature and not LipsyncFeature or TextToSpeechFeature. Have you made any other changes along with speech and lipsync? I have seen the above error occur when the object the PointOfInterestFeature is targeting has an invalid transformation matrix. I would try removing the PointOfInterestFeature temporarily to see if your lipsync will work without it. If that is the case we can close this out and open a separate issue specific to point of interest for further investigation.
from amazon-sumerian-hosts.
Related Issues (20)
- Help for deploy in a public webserver HOT 2
- AWS-Infrastructure-Setup.md Documentation Is Very Obsolete HOT 1
- How did you create the libsync.glb file? HOT 1
- unable to get examples to work HOT 2
- Mouth Does Not Move With Speech HOT 7
- Add support for BabylonJS v5 HOT 13
- Speech animation, gaze tracking, face shape break with BabylonJS v5 HOT 1
- Custom Avatar Models for BabylonJS HOT 1
- LexRuntimeV2 recognizeUtterance Response HOT 2
- Cannot create and run "production" builds HOT 1
- Audio not working with BabylonJS Demo on iOS devices HOT 1
- Editing glTF assets with DCC Tools (import/export) degrades content. HOT 5
- How to develop a custom Amazon Sumerian Host avatar? HOT 9
- PlayCanvas host HOT 3
- Remove webpack, use vanilla ES Modules to simplify the project. HOT 2
- loading forever on mac HOT 3
- -
- replace Karma with @web/test-runner
- GET /_next/webpack-hmr 404 HOT 1
- Babylon types not visible
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-sumerian-hosts.