Why Can a Different Lip Movement Change What We Hear?

When we listen to someone speak, it feels as though the ears are doing the work alone. Vision is involved as well. In McGurk and MacDonald's classic 1976 experiment, the sound “ba” was dubbed onto a face articulating “ga”. Many participants reported hearing a third sound, such as “da”. The acoustic signal had not changed; what changed was the brain's combined interpretation of the two cues. This is known as the McGurk effect.

The finding shows that speech perception is not a process in which the ears first deliver a complete answer and the eyes merely add detail. The brain uses both acoustic information and movements of the lips and jaw to infer what a speaker most likely said. In face-to-face conversation, this integration is usually helpful, especially when speech is noisy or unclear. The same process can also create a reliable illusion in an experiment.

The McGurk effect is not a rule that vision always defeats hearing. Its strength varies with the speaker, listener, language experience, attention, and the timing of sound and image; some people show little effect at all. A single demonstration cannot diagnose a problem with hearing, attention or comprehension. The more careful conclusion is that heard speech is a perception formed from several sensory cues in a particular context, not a word-for-word copy of sound waves.

https://doi.org/10.1038/264746a0