Baldi and Timo are animated talking heads whose speech synthesis is so accurate it can be read by skilled lip readers and students with hearing loss.
Baldi's skill—recently transferred to Timo—has evolved with the help of UCSC Psychology Professor Dom Massaro, a dedicated team of Perceptual Science Lab researchers, and ongoing funding from the National Institute of Deafness and Communicative Disorders.
If you made Baldi's skin transparent you would see that, at his core, he's a true 3-dimensional wire frame model. What you wouldn't see is the authoring engine comprised of software tools that enable even non-technical people to control his facial movements, emotions, intentions, and appearance.
We've found it useful to describe Baldi as an electronic puppet with over 100 strings. Sixty-one of these "strings" can be used to control the movement of jaws, tongue, lower and upper lips, cheeks, muscles, eye movements, mouth widening, air flow, breath, and more. Professor Massaro and his team have established ideal set mathematical parameters for each of these 61 "strings" enabling Baldi, his protégé Timo, and other animated tutors to accurately mimic the real-life face of a speaking human. Each "string" comes with an ideal setting but can be adjusted to create individual uniqueness and add emphasis to specific words and emotions.
But...how does he know what to say? With the help of the automated software tools mentioned above, Baldi can be programmed to say anything. His "visible speech" is driven by either 1) a recorded voice or 2) text that is automatically converted in real-time into a synthesized voice. Baldi's articulation will align automatically with either type of input.
For more detailed information about the technology, please refer to Dr. Massaro's paper entitled, "Symbiotic Value of an Embodied Agent in Language Learning - HTML page or 148KB Adobe PDF."