McGurk Effect (Audio-Visual Illusion)

Hearing with your eyes

In the classic version, a video shows a face saying "ga" while the soundtrack plays "ba". Watching and listening together, most people hear "da", a syllable that's in neither the sound nor the picture. Close your eyes and it turns straight back into "ba".

The effect needs moving video with sound, so the diagram above can only describe it. To experience it, watch the BBC's short McGurk demonstration with the sound on. Watch the speaker's mouth the first time, then play it again with your eyes closed and listen for the change. The BBC clip uses different syllables from the "ba"/"ga" example, but it works the same way.

What makes the McGurk effect striking is that you can't switch it off. Knowing the sound is "ba" doesn't help: as soon as you watch the mouth, you hear something else.

The brain splits the difference

Each consonant comes with a visible mouth movement. "Ba" needs the lips pressed together; "ga" is made at the back of the mouth with the lips open, and most people can tell the two apart by lip-reading alone. We learn to speak and listen in a world where sound and lip movements almost always agree, so the brain doesn't treat vision as a backup for when the sound is unclear. It combines both all the time.

When the two conflict, it doesn't simply pick one. It looks for the syllable that best fits both, and "da" is that compromise: it's made with the tongue just behind the teeth, between the closed lips of "ba" and the back-of-the-mouth "ga", so it isn't badly inconsistent with either the sound or the picture. The result is a sound that nobody produced.

The brain combines senses like this whenever one alone is ambiguous, giving more weight to whichever is more reliable for the task. Speech is an unusually tight case because everyone lip-reads a little, mostly without realising it, which is why it helps so much to see someone's face in a noisy room.

Not everyone hears it

The effect is strong but not universal, and people differ a lot. When Debshila Basu Mallick and colleagues tested 165 adults on twelve different McGurk videos, some never heard the fused syllable and others heard it every time. People retested a year later mostly scored much as before, so the difference looks like a stable trait rather than chance (Basu Mallick et al., 2015). The videos varied too: some produced the illusion far more often than others.

Language matters as well. Kaoru Sekiyama and Yoh'ichi Tohkura found that Japanese listeners hearing clear Japanese syllables showed much weaker effects than English speakers typically do, although the effect came back strongly when noise was added to the sound (Sekiyama & Tohkura, 1991). One explanation is that listeners who lean less on lip-reading in everyday conversation give the visual signal less weight unless the sound is hard to make out.

Discovered by accident

The effect is named after the psychologist Harry McGurk and his research assistant John MacDonald, who reported it in Nature in 1976 under the title "Hearing lips and seeing voices". They were studying how infants perceive speech, and according to their own account, the mismatched video came about by accident when the wrong syllable was dubbed onto a recording. They heard the fused sound themselves before realising what it meant, and the paper has since become a cornerstone of research on how the senses combine.