Disney Research Lab Comes Up With Automated Re-Dubbing: How To Put Words In Your Mouth
Disney Research Lab has introduced a new technology that can automatically re-dub detecting your lip movements and facial expressions. Visemes, the visual aspects of speech are detected in videos and then they are mapped to several phoneme progressions to be synchronized.
Disney has outdone itself, first it were the 3-D printers that constructed hug-gable objects, then the robots and spinning tops and now this. The fascinating technology introduced by the company is named, in the paper written by Sarah Taylor and Iain Matthews Disney Labs Pittsburg, ‘Visually Consistent Acoustic Redubbing’.
The new method introduced by the media giant differs from the traditional methods of dubbing and speech recognition. Whereas, most softwares read the static lip movement, only this one in specific maps the facial expressions as well.
“This work highlights the extreme level of ambiguity in visual-only speech recognition,” said Taylor, who is also lead author of the study. “Dynamic visemes are a more accurate model of visual speech articulation than conventional visemes and can generate visually plausible phonetic sequences with far greater linguistic diversity.”
A demo of a person saying ‘clean swatches’ could be easily dubbed with 9000 possible different phrases. Not all phrases make sense, but it can be replaced with anything that aligns with the visuals including ‘need no pots’ to ‘like to watch you’.
Another such app which is quite famous nowadays, Dubsmash, fulfills almost the same purpose. It provides users with an array of audio clips, they can then shoot a selfie video while saying the same lines and the app automatically generates a video with the dubbed audio.
This technology can benefit the film industry immensely, as it can be used in the fields of automated speech dubbing for audiences that speak in a different language.