Retinex Theory

When I was young I learned that we can see objects because of the light reflected by objects.

Sunlight (or let’s say white light) contains all wavelengths of color. When this white light falls into a green surface like in the figure, the wavelength corresponding to green light is reflected. Then, when these green light rays fall into our eyes, we see the color of the object.

This is the basic theory we learned in school. Now let me tell you an interesting fact. If you look at the figure at the top, you can see the beautiful Loisach River in Germany with mesmerizing green scenery. But actually, this image contains only gray and red shades. When you go to a musical concert, the stage is illuminated with different colored lights, you can see the actual colors of the objects (you don’t say a blue man is singing a song). Another example is when in the evening you can distinguish colors even though it is a reddish environment (sometimes get confused, but most of the time you know the actual color). This is called “color consistency”. Color constancy allows for humans to interact with the world in a consistent or veridical manner and it allows for one to more effectively make judgments on the time of day.

This contradicts my basic understanding of how I see. If I receive wavelengths of red light, I cannot see other colors. But, I can understand that the image contains different colors. In this article let’s explore and realize why this happens.

Before we proceed let me give a brief overview of human anatomy related to vision.

As you can see in the figure below, the light entering the eye falls to the region called Retina. This is a thin layer of tissue that lines the back of the eye on the inside. The purpose of the retina is to receive light that the lens has focused, convert the light into neural signals, and send these signals on to the brain for visual recognition.

Let’s see whether the solution to our problem lies here. First, let’s understand how the retina converts the light into neural signals.

In humans, this light to neural signal conversion happens using two types of photoreceptors, “cones”, and “rods”. There are three types of cones in humans that are sensitive to Red, Green, and Blue. More formally they are called L for long-wave (Red), M for middle-wave (Green), and S for short-wave (Blue) sensitive cones. (The rods are capable of functioning in lower light conditions better than cones. ~like night vision mode of cones. Let’s focus more on cones for now.)

The light falling on the retina determines the L, M, S cone responses, namely, the “quanta catch” of the cones. The cone quanta catch is the important transition from the physics of light to the physiology of vision. In summary, the eye catches the light and convert into neural signals based on the wavelength of the light entered to the eye. This corresponds to what I learned in school.

Now let’s do a simple experiment. There is a green and a red circle on the black surface like in the figure below. Then we illuminate them using white light. A filter is used to absorb the Middle waves for the green circle. Now the light falls to green circle as Long and Short waves (magenta arrows). Similarly, for the red circle, we use a filter that absorbs Long waves (cyan arrows).

The light coming to the eye (yellow arrows) is the product of illuminating light (magenta/cyan arrows) and the reflectance (green/red) of the circle. This is shown in the first 3 graphs illustrated below. The integrals of the scattered light using the three-cone spectral sensitivities determine the L, M, S quanta catch values. Interestingly they are both same for the above experiment even though we have different colored circles. i.e. the eye senses them as the same color objects. (If we do an experiment by asking people to compare the color of the object, they will say that the both circles have the same color.)

Now we know that our eye does the job of transforming light wavelength into neural signals properly.

Let’s explore what happens to the neural signals are sent to the brain.

After signals are sent to the brain, the primary visual cortex in the brain computes local ratios of cone activity. There are specialized neurons called double-opponent cells. They compute both color opponency and spatial opponency. The existence of these cells in the primate visual system was proven by selectively activating single cone classes at a time, so-called “cone-isolating” stimuli. From this information, the visual system attempts to determine the approximate composition of the illuminating light. This illumination is then discounted in order to obtain the object’s “true color” or reflectance: the wavelengths of light the object reflects. This reflectance then largely determines the perceived color.

There is a clear interconnection between the retina and visual cortex that gives this color consistency that we observe. This was first put forward by Edwin H. Land.

He realized that, even when there were no green or blue wavelengths present in an image, the visual system would still perceive them as green or blue by discounting the red illumination (Like the first image on top). In 1977, Land wrote a Scientific American article that formulated his “Retinex theory” (formed from “retina” and “cortex”, suggesting that both the eye and the brain are involved in the processing) to explain this effect.

Color constancy works only if the incident illumination contains a range of wavelengths. The different cone cells of the eye register different but overlapping ranges of wavelengths of the light reflected by every object in the scene. The important is that in complex scenes, a particular quanta catch can appear in any color: red, green, blue, yellow, white, or black. (Like shown above we can’t distinguish green and red circle, because it is the only wavelength falls into the eye)

Land did several experiments to show how color consistency works. So let’s go through them to understand the Retinex theory!

Mondrian Experiments

Land used two identical Mondrians (abstract art like the paintings by Piet Mondrian) made of color papers, and three different, non-overlapping spectral illuminants (long-, middle-, and short-wave visible light).

In this experiment, observers reported the colors of patches in the Mondrians. We need to focus on the red and green circle patches. Land adjusted the illuminant mixtures of light from the two sets of three projectors (Instead of filters like we used above. Now he has more control over the illuminant light).

He adjusted illuminant light so that left-green and right-red circles had equal quanta catches by the L, M, S cones. 🤔

In this scene, observers reported that the red paper looked red and the green paper looked green despite the identical cone quanta catch. 😱

Land repeated this experiment with all the Mondrian papers. A constant L, M, S quanta catch could generate any color sensation. The presence of the complex scene introduced more information to the visual system.

The scene’s spatial content stimulated vision’s spatial image processing mechanisms to generate color constancy. Post-receptor visual processing plays a dominant role in color appearance in real scenes.

In the following figure, there are two sets of nine red squares that have the same reflectance and appear the same (top left). However, if these red squares are surrounded by yellow and blue stripes, they look different (top center): the left red squares fall on top of the yellow stripes, and the right ones on the blue stripes. The left squares appear a purple-red, while the right ones appear a yellow-orange. In other words, the left squares appear more blue and the right ones more yellow.

In the LMS separation (decompose the image into wavelengths correspond to L, M and S waves) you can see that in the right side red dots, its more red, green, and less blue. Similarly, you can deduce why the left side red dots are purple-red.

If you can remember the color inconsistency where some see the color of the dress as blue (like in right), some as a yellow dress (like in left). Now you can know the reason for this. (basically where you at when looking at it; the surrounding of your phone if you are looking through the phone)

Another 2 examples are shown below. The colors in cells A and B appear different even though they are the same color. On the right side, you can see a person holding colored papers. Two yellow papers that he’s holding are actually not yellow.

Even if these type of illusions are consequences of color consistency that happens in our visual cortex, it is very important for humans to understand the scene.

Now you know how humans correctly identify colors in different illumination conditions. Retinex theory is all about that.

Here comes the important question. Usual cameras do the work of the eye. They catch the light and store the signals in a storage device. Now how to do the work of the visual cortex. How to give computers the ability of humans. How to make computers understand an image with different illumination conditions?

I will explain it in a different article.

Thank you for reading 😍🤓