Sound and Animation synching
Shri

Gio,

I was wondering if you had any suggestions on how to synchronize a sound with an animation. For example, if a character were talking, how would you synch the sound with the mouth moving ? Is this something that you would consider adding to the engine in the future ? or is this a big ask ?

cheers - shri

1 Comment
Gio

Hi Shri

I'm not sure I understand how this would work. You mean that you have one single "talk" animation, and you want to play it and stop it in sync with a single audio file that contains some speech and pauses between words or sentences?

If so, provided that WebAudio is supported on your target platform, I think you could achieve that by monitoring the audio level of the file you are playing through the web audio interface. As you know (I remember you made a WebAudio demo some time ago with this stuff), this involves creating an Analyser node to get the signal level in real time.

In fact if you want to do this properly, you don't want to get the global signal level, you want to look at some frequencies only: those that typically contain human speech, i.e. about 20Hz to 2Khz. This is actually pretty straightforward with WebAudio.

Anyway, once you have the volume level, you need to decide what you want to do with the animation. I think the logic would be something like this:

  • If the animation is not playing, and the volume level is high, start playing
  • If the animation is not playing, adn the volume level is low, do nothing
  • If the animation is playing and the volume is low, change the animation speed so it can finish asap
  • If the animation is playing and the volume is high, do nothing.

 

Now if you had several animations, say one for each vowel, then this would get very interesting with some hardcore signal processing :)

I'm not ruling out adding something like this to Wade in the future but probably not in the core of the engine. For version 4.0 we are working on ways to easily add snippets of code to your poject without putting thing in the core wade.js which we want to remain small.

Post a reply
Add Attachment
Submit Reply
Login to Reply