Predictive Processing and the Brain's Shortcuts

For most of the history of vision science, the standard picture ran from the bottom up: light hits the eye, signals climb through the visual system, and near the top a recognisable image appears, like an assembly line running in one direction. Predictive processing turns that picture round. It treats the brain as a prediction engine that constantly generates a model of what it expects to see, and uses the signals from the eyes mainly to check and correct that model rather than to build perception from scratch (Clark, 2013).

Unconscious inference

The idea goes back to the nineteenth-century physicist and physiologist Hermann von Helmholtz. He argued that perception is "unconscious inference": the eyes never deliver an unambiguous picture, so the brain infers the most likely cause of what it receives, drawing on past experience, much as a detective reasons from partial evidence (Helmholtz, 1867). He lacked the mathematics available today, but the core point, that perception is inference rather than transcription, anticipated the modern theory.

Today the idea is usually expressed in terms of Bayesian probability. The brain holds "priors", expectations about the world built from experience, and combines them with the incoming evidence to reach its best current guess about what's out there (Kersten, Mamassian & Yuille, 2004). Crucially, the evidence isn't always trusted equally. When it's noisy or ambiguous, the prior dominates; when it's clear, perception follows the evidence. Karl Friston has developed this into a broader theory in which perception, action and learning all serve to reduce the brain's prediction errors about its surroundings (Friston, 2005).

Prediction error

One surprising claim of the framework is that what travels up the visual hierarchy isn't mainly the raw image but the error: the mismatch between what higher areas predicted and what lower areas detected. When the prediction is right, there's little left to send. When it's wrong, the error travels up and the model is updated. Rajesh Rao and Dana Ballard showed that a network working this way reproduces some puzzling properties of real neurons in the visual cortex (Rao & Ballard, 1999). It also gives a role to the many connections running from higher areas back down to V1, described in the role of the visual cortex: they can be seen as predictions sent down to be checked against what arrives from the eyes.

Why this explains illusions so well

If perception is prediction shaped by expectation, illusions are what you'd expect when an image triggers a strong prior that happens to be wrong.

In the checker shadow illusion, the brain's strong expectation that shadows darken surfaces makes it judge the shaded square as lighter than the light from it suggests. In the hollow face illusion, the expectation that faces bulge outwards is strong enough to overrule direct evidence that the mask is hollow. Motion gives a particularly neat example. Yair Weiss, Eero Simoncelli and Edward Adelson showed that many motion illusions follow if the brain assumes, other things being equal, that things tend to move slowly. When a moving pattern is faint or low in contrast, the evidence is weak, the slow-motion prior wins, and the pattern looks slower than it is (Weiss, Simoncelli & Adelson, 2002). The Necker cube fits too: with no evidence favouring either of two equally likely interpretations, perception alternates between them.

Expectation can also be measured directly. The duck-rabbit tends to look like a rabbit at Easter, and the site's spinning dancer poll tests a specific prior: our preference for seeing figures from above.

A leading theory, still being tested

Predictive processing is one of the most influential ideas in neuroscience and cognitive science today, applied to perception, movement, mental illness and even consciousness. It isn't settled. Researchers still debate how priors are learned, how they're implemented in neurons, and how far the framework can be stretched before it stops making testable predictions. As a way of understanding illusions, though, it explains something the older bottom-up story didn't: why the brain would ever produce a confident, vivid and wrong perception rather than simply failing to see. For the wider question of why perception and reality come apart at all, see why do optical illusions happen.

Sources and further reading

1. Helmholtz, H. von (1867). Handbuch der physiologischen Optik. Leopold Voss. Find this reference on Google Scholar
2. Rao, R. P. N., & Ballard, D. H. (1999). Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1), 79–87. Find this reference on Google Scholar
3. Weiss, Y., Simoncelli, E. P., & Adelson, E. H. (2002). Motion illusions as optimal percepts. Nature Neuroscience, 5(6), 598–604. Find this reference on Google Scholar
4. Kersten, D., Mamassian, P., & Yuille, A. (2004). Object perception as Bayesian inference. Annual Review of Psychology, 55, 271–304. Find this reference on Google Scholar
5. Friston, K. (2005). A theory of cortical responses. Philosophical Transactions of the Royal Society B, 360(1456), 815–836. Find this reference on Google Scholar
6. Clark, A. (2013). Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3), 181–204. Find this reference on Google Scholar