How the Brain Processes Visual Information

Seeing feels effortless: you open your eyes and the world is there, already sorted into objects, distances, colours and movement. That effortlessness hides one of the most demanding jobs the brain does. A large share of the cortex is involved, and the signal passes through several stages, each of which turns light into something closer to knowledge.

From light to signals

Vision starts as physics. Light enters the eye through the cornea and lens and lands on the retina, a thin sheet of tissue at the back of the eye containing photoreceptors: cones, which handle colour and fine detail in good light, and rods, which handle dim light. Neither "sees" anything. They convert light into electrical signals.

The retina is already processing before anything leaves the eye. Its layers of bipolar, horizontal, amacrine and ganglion cells compare each receptor's output with its neighbours'. Many ganglion cells have a centre-surround arrangement: light on the centre of their patch excites them, light on the ring around it inhibits them, or the other way round. The result is that the retina emphasises edges and change rather than raw brightness, a theme that runs through the whole visual system. It's also why the simultaneous contrast squares look different: brightness is registered relative to the surroundings from the very first stage.

The relay through the thalamus

About a million fibres from each eye form the optic nerve. Most of them don't go straight to the cortex: they stop first at the lateral geniculate nucleus (LGN) in the thalamus, which sorts the signals by eye, position and type, and filters them under the influence of feedback from the cortex. Even at this stage, vision isn't a one-way camera feed; incoming data is already being shaped by what the brain expects, an idea at the centre of predictive processing.

Into the cortex: "what" and "where"

From the LGN, signals reach the primary visual cortex (V1) at the back of the brain. David Hubel and Torsten Wiesel discovered that cells here respond to simple features such as edges at particular angles (Hubel & Wiesel, 1962), and that the different features of an image are handled by partly separate channels (Livingstone & Hubel, 1988). The visual cortex page goes into more detail.

From V1, processing splits into two broad streams. The ventral stream, running towards the temporal lobe, deals with what things are: objects, faces, written words. The dorsal stream, running towards the parietal lobe, deals with where things are and how to act on them: location, movement and guiding the hand. Leslie Ungerleider and Mortimer Mishkin first described this as a "what" versus "where" division (Ungerleider & Mishkin, 1982), and David Milner and Melvyn Goodale later recast it as "what" versus "how" (Goodale & Milner, 1992). It helps explain why some brain injuries leave people able to describe an object but not reach for it accurately, or the reverse. The Ebbinghaus illusion has been used to test whether the two streams can be fooled separately.

Not a camera

At every stage the system compresses, filters and interprets rather than records. Each eye has a blind spot where the optic nerve leaves, yet you never see a hole, because the brain fills it in. Your eyes jump several times a second, yet the world doesn't. Depth, which doesn't exist on the flat retina, is rebuilt from clues such as perspective and relative size, which is exactly what the Ponzo illusion, the Ames room and anamorphic art exploit.

Where illusions fit

Every illusion on this site targets some stage of this pipeline. Some work on early contrast processing, like the café wall. Some work on how the cortex combines cues to judge depth and lightness, like the checker shadow. Some work on how the brain groups shapes into objects, the subject of Gestalt principles of perception. None of them fools a broken system. They show a system built for speed and useful guesses, doing what it evolved to do under conditions designed to expose its shortcuts. That's the starting point for understanding why optical illusions happen.

Sources and further reading

1. Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. Journal of Physiology, 160(1), 106–154. Find this reference on Google Scholar
2. Ungerleider, L. G., & Mishkin, M. (1982). Two cortical visual systems. In D. J. Ingle, M. A. Goodale, & R. J. W. Mansfield (Eds.), Analysis of Visual Behavior, pp. 549–586, MIT Press. Find this reference on Google Scholar
3. Goodale, M. A., & Milner, A. D. (1992). Separate visual pathways for perception and action. Trends in Neurosciences, 15(1), 20–25. Find this reference on Google Scholar
4. Livingstone, M., & Hubel, D. (1988). Segregation of form, color, movement, and depth: Anatomy, physiology, and perception. Science, 240(4853), 740–749. Find this reference on Google Scholar