Special Section on Retinex at 50

Analysis of retinal and cortical components of Retinex algorithms

[+] Author Affiliations
Jihyun Yeonan-Kim, Marcelo Bertalmío

Universitat Pompeu Fabra, Departament de Tecnologies de l’Informació i les Comunicacions, Barcelona, Spain

J. Electron. Imaging. 26(3), 031208 (May 17, 2017). doi:10.1117/1.JEI.26.3.031208
History: Received October 24, 2016; Accepted April 26, 2017
Text Size: A A A

Open Access Open Access

Abstract.  Following Land and McCann’s first proposal of the Retinex theory, numerous Retinex algorithms that differ considerably both algorithmically and functionally have been developed. We clarify the relationships among various Retinex families by associating their spatial processing structures to the neural organizations in the retina and the primary visual cortex in the brain. Some of the Retinex algorithms have a retina-like processing structure (Land’s designator idea and NASA Retinex), and some show a close connection with the cortical structures in the primary visual area of the brain (two-dimensional L&M Retinex). A third group of Retinexes (the variational Retinex) manifests an explicit algorithmic relation to Wilson–Cowan’s physiological model. We intend to overview these three groups of Retinexes with the frame of reference in the biological visual mechanisms.

Figures in this Article

The Retinex color theory was postulated to account for a series of observations that Land made on color constancy.1 He and his colleagues reported several experiments in which they presented color pictures comprised of color patches reminiscent of the paintings of Mondrian under various trichromatic spectral combinations of illuminant, that is, different amounts of the long-wave (red), middle-wave (green), and short-wave (blue). They examined the color appearance of a particular patch in the Mondrian picture. Two different Mondrian pictures were viewed, each containing a target patch. The two target patches had different spectral reflectances, say, white and green. They adjusted illuminants under which the Mondrian pictures are observed such that the target patches with different reflectances in the two pictures emitted the same spectral radiance. Despite the two patches having equal spectral radiance, the white and green patches did not appear the same. Rather, they were perceived as white and green, respectively, thereby demonstrating color constancy.

Similar results were obtained in a different experimental setup, in which an observer viewed one Mondrian-picture under varying combinations of illuminant triplets and determined the color of the patch by choosing the matching color from the Munsell chart. In each run, the illuminant was adjusted so that the patch always sent the same radiance to the eye regardless of its reflectance. Despite the target patch having different spectral radiances, the observer matched the patch to the closest color in the Munsell chart in all cases, e.g., green patch to the green color and so on.

Land’s experiments revealed that color sensation cannot be predicted based on a simple analysis on the spectral radiance of the incoming light from the scene to the eyes. What, then, determines color sensation? From what aspects of a scene can color sensation be predicted? The Retinex theory sought to solve this problem.

The fundamental Retinex color principles were established during the early development of the theory.15 First, color sensations correlate with the lightness of independent long-, medium-, and short-wavelength spectral channels (channel independence). Second, the lightness of an image area in each waveband channel depends on a complex relationship of the area and its surroundings (spatial interaction).

The prototype Retinex algorithm by Land and McCann,3 referred to as L&M Retinex (throughout this article, we will use the conventional Retinex names outlined in Ref. 6), describes the computational process to realize the principle of spatial interaction by computing the radiance-ratio product along paths that go through the image. This process is applied to each of the triplet of the color channels (channel independence). The algorithm iteratively computes the output array by assigning an arbitrary initial value 1.0 radiance-ratio product for all the pixel locations in the image and updating the value at each pixel as a path passes through. The path makes spatial comparisons by taking ratios of scene radiances at the edges and uses products of these ratios to make long distance comparisons. A reset procedure (a ratio-product value larger than 1.0 is set to 1.0) is used for scene normalization to maxima. This path procedure was repeated for multiple paths, providing a way to model appearance. The number and lengths of the paths determine the balance between local-maxima and global-maximum scene normalization. Path parameters were optimized by observer matching data that provided a quantitative goal for this model of vision.

Succeeding and reinventing the original Retinex formulation, numerous Retinex methods were developed in the past decades (see Ref. 6 for a comprehensive review). All the newer Retinex algorithms maintained the basic principle of channel independence. On the other hand, the one-dimensional (1-D)-path-based spatial interaction of the prototype L&M Retinex3 was quickly replaced by a two-dimensional (2-D) operation for most of the subsequent Retinex-family algorithms [Frankle and McCann,7 McCann 99,8,9 Land’s designator Retinex,10 NASA retinex,11,12 gamut Retinex,13 Sobol Retinex,14,15 Kotera Retinex,16 Variational Retinex,1720 kernel-based Retinex (KBR);21 cf. Milano Retinex22,23], but the strategical details diverged substantially across them. In addition, some portion of the Retinex family removed the reset operation (or any other white-anchoring mechanisms) and either adopted the grayworld hypothesis (the algorithmic assumption that the global mean of any image is gray; as in designator Retinex, variational Retinex) or not (e.g., NASA Retinex).

That some of the basic principles of the prototype L&M Retinex are altered or eliminated in some Retinex variations leads to the question: are all these Retinex successors designed to accomplish the same processing goal, which, for the L&M Retinex, was predicting color sensation? Is there a way to interpret different Retinex designs in an integrative perspective?

In the search for the answer, we noticed that different Retinex variations have comparable neural mechanisms in the human visual system that provides a comprehensive framework. Land and McCann’s3 Retinex aimed to predict color sensation, which is the end result of the processes consisting of the human visual system. Thus, the visual system mechanism is considered the “ground truth” of the Retinex theory, as Land named it with the compound word of retina and cortex.

In the 1980s, Land published some works that modified the prototype L&M Retinex formulas (the designator Retinex).10,24 These ideas further inspired Jobson, Rahman, and Woodall to develop another line of 2-D Retinex (NASA Retinex)11,12 (followed by Kotera Retinex).16 As we will soon detail, this line of Retinex implementation shows an explicit connection to Retinal processing. The designator and NASA Retinexes do not involve the reset.

In 1975, Frankle and McCann Retinex began the implementation of 2-D L&M Retinex,7 which evolved into McCann 99 Retinex8,9 (continued to gamut Retinex13 and Sobol Retinex14,15). This Retinex family preserves all the prototype L&M Retinex principles. At a glance, the relationship of 2-D L&M Retinexes with a neural mechanism may not be clear. But we will point out that the shifted-array comparison approximates orientation-dependent processing in the visual primary visual cortex. These Retinexes maintain the spatial interaction schemes of the original L&M Retinex3 including the reset and average operations.

There is an another line of 2-D Retinexes, based on variational methods, that are associated with image processing problems (contrast and color enhancement and histogram equalization25 and reflectance estimation1720). They share the same algorithmic structure with a physiological model of cortical process, the Wilson–Cowan model,26 but the relationship of their functional structure of spatial interaction to the visual system mechanisms has yet to be established. Variational Retinexes, similar to the designator and NASA Retinexes, do not incorporate the reset operation.

Here we will line up the above three families of Retinexes alongside the retinocortical neural mechanisms, the different ground truths used by different Retinexes. Our discussion is concentrated less on the mathematical details of the different Retinex versions from each of the Retinex families and more on the history and the implications of the pivotal versions that manifest connections with neural mechanisms: the multiscale NASA Retinex,12 McCann 99 2-D L&M Retinex, and the variational model by Bertalmío et al.21

We first provide the basic information on the retinocortical processing architecture as a guideline. The visual brain is a complicated machinery, and we do not intend to cover all the neurophysiological details. It is sufficient for our context to overview the basic spatial processing properties of the retina and the primary visual cortex.

The gist of spatial processing in the retinocortical stream is the push-pull spatial interaction and the multiscale structure. The precise ways in which the spatial interaction occurs differ between the retina and the cortex, and this determines what type of information is handled at each stage.

Retinal Push-Pull Spatial Interaction

Light reaching the retinal surface triggers the phototransduction process of photoreceptors, which transform light energy into neural activity. From this very first stage of the visual processing chain, the human visual system discards any redundant information by coding the “difference” of the light signals at one point of the visual scene from that of nearby points.

In the retina, the spatial interaction is accomplished through a regulatory mechanism called “lateral inhibition” (see Ref. 27 for a comprehensive review). When a set of photoreceptors is activated by light projected to some part of the visual field, several regulatory interneurons (named horizontal cells) around the photoreceptors are consequently activated that are connected back onto the photoreceptors and inhibit the activities of the photoreceptors. Thus, when the photoreceptor responses increase (push), the horizontal cells roll their responses back (pull). The reason that the inhibition is “lateral” is because the receptive-field of a horizontal cell (i.e., the pull area) is larger than the receptive-field of a photoreceptor (i.e., the push area), so an activation of a single photoreceptor would result in the inhibition of the neighboring photoreceptors to which the same horizontal cell makes an inhibitory connection.

As the outcome of this push-pull regulation, the retina codes a light intensity at a certain point of a scene as the difference of the intensity value at the point (center) from the mean light intensity surrounding the point (surround; thus referred to center-surround processing). The center-surround processing is commonly represented as a difference-of-Gaussian (DOG) following the estimated profile of the ganglion cell receptive-field.2832 In other words, the retinal spatial processing is abstracted as an isotropic (i.e., radially symmetric) spatial filter that is the difference of two Gaussian filters, one with a smaller standard deviation (push) and the other with a larger standard deviation (pull).

Orientation-Dependent Cortical Push-Pull

The orientation-sensitivity is the signature spatial processing characteristic of the primary visual cortex.3335 Different cells in this brain region have their own “preferred” or “tuned” orientation in the incoming light pattern. For example, assume that we measure the electric activities of a single cell in the cortex of a primate when the animal is viewing different orientations of bars. A cell may show the most rigorous activity when the animal sees a vertical bar but may remain silent or inhibited upon seeing a horizontal bar. This particular cell is tuned to vertical (0 deg) orientation.

The orientation preference of the cells in the primary visual cortex of primate species and humans is well documented. Here, the cells are aligned based on their orientation preference, orderly from 0 deg to 360 deg, for various spatial frequencies (resolution; see next section) for each location in the visual field. This orientation-dependent structure is substantially different from the isotropic spatial representation of the retinal ganglion cells.

There is a type of push-pull regulation at the primary visual cortex that occurs with respect to the orientation preference. Taking the example above, the vertical-tuned cell activity (push) is regulated by the inhibition from the cells tuned to nearby orientations (pull). The push-pull regulation in the primary visual cortex thus normalizes the cell activities across orientations. In other words, it codes the “difference” of a particular orientation information in the incoming light pattern from close-by orientations.

Retinal and Cortical Multiresolution Structure

In addition to the orientation-preference, the primary visual cortical cells exhibit spatial-frequency-preference;3335 that is, a cell responds more strongly to a certain spatial-frequency, or resolution, of stimulus over other resolutions. Thus, the organization of the primary visual cortex is such that there exist cells with different combinations of spatial-frequency and orientation sensitivity.

On the other hand, there has not been much discussion on any multiresolution structures of the retina. Most, if not all, of the studies involving lateral inhibition up-to-date have assumed a single-surround DOG. This is based on earlier physiological evidence on the retinal ganglion cell RF, which is often referred to as the “classic” RF.28,3032 However, Yeonan-Kim and Bertalmío36 pointed out that there is accumulating evidence that the spatial profiles of the retinal regulatory interneurons are comprised of multiresolution components3739 and that the surround part of the ganglion cell exhibits an “extra-classic” suppressive field that subtends much beyond the width of the classic RF.4042 This means that the retinal spatial processing incorporates a multiresolution surround structure.

The first group of Retinexes of our concern includes the Land’s designator Retinex and NASA Retinex. The spatial interaction forms of this family manifest a straightforward correspondence to the neural structure of the retina.

Designator Retinex by Land

In 1983, Land proposed a new process of Retinex computation (designator Retinex) distinguished from the L&M family of Retinex. He proposed the concept of designator,24 by which he meant a local average of the ratio product (i.e., local lightness) around a given point in the image of interest, (xn,yn), for each color wavelength channel. His initial idea was to compute the ratio product along a path from (xn,yn) to another point in the image, (xm,ym), repeat this ratio-product computation for various image points (xm,ym)s around (xn,yn), average across all the ratio-products obtained between (xn,yn) and each of the (xm,ym)s, and use this average as the Retinex output for the point (xn,yn). Designator Retinex involved thresholding (discarding any ratio on the path that is smaller than a certain value) but not reset.

In 1986, Land described a designator Retinex using 2-D computations instead of involving the path:10

“The designator is the computed numerical measure on one waveband of the lightness seen as part of the whole field of view. Previous retinex techniques have involved some kind of comparison between the flux (on one waveband) coming to the eye from a point on the object and flux (on that same waveband) arriving from points in remote, as well as contiguous, areas. These comparisons involve edges, gradients, thresholds, and pathways, and provide the average of the relationships between a given point and a large number of other points in the field of view … Keeping the same criteria, the new technique, instead of utilizing an average of these relationships, compares the flux from the point of interest to an average, weighted in an unusual way, of the fluxes from all points in the field.”

Here, Land suggests comparing a point in the image to the average radiance around the point, rather than computing the radiance ratio of a point to each of the points in the neighborhood followed by averaging.

This is reminiscent of the neural structure of retina that we introduced in Sec. 2.1 (despite the fact that Land commented in his 1983 paper that color coding in the cortical area V4 may be of further consideration in relation to his idea)24 when we make an analogy between a point in the image and a single photoreceptor on the retinal surface. The retinal push-pull regulation and lateral inhibition is such that a horizontal cell averages across responses of multiple photoreceptors (i.e., multiple points in the image) and this response average regulates, or inhibits, the responses of each photoreceptor (i.e., a point in the image).

Retinal Lateral Inhibition and NASA Retinex

Land may not have further explored his idea on the 2-D implementation of designator Retinex,10 but a similar concept was elaborated by other researchers, who developed a center-surround type of Retinex processing comparable to retinal lateral inhibition.11,12,4345 Lateral inhibition codes light intensity at a certain point of a scene as the difference between the intensity value at the point (center) and the mean light intensity surrounding the point (surround; thus referred to as center-surround processing).

The NASA Retinex proposed by Jobson, Rahman, and Woodell11 is one of the Retinex versions inspired by retinal processing, which the authors called “center/surround” Retinex.

The center/surround NASA Retinex takes the form Display Formula

Ri(x,y)=logIi(x,y)log[F(x,y)*Ii(x,y)],(1)
where i indicates the i’th trichromatic spectral band of the image I with (x,y) denoting the pixel position and * stands for the convolution operation. F(x,y) is a 2-D function for surround, for which the authors used either a Gaussian or an exponential function.

Jobson et al.11 noted that this is the typical DOG representation of retinal and LGN ganglion cell RFs28,3032 (see Sec. 2.1) and that Eq. (1) approximates the ratio of the center pixel radiance with respect to the average radiance of the surrounding pixels as long as the source illumination is uniform across the image.

The above NASA Retinex algorithm is referred to as a single-scale Retinex (SSR),11 in contrast to a multi-scale Retinex (MSR)12 that the same authors subsequently developed to overcome the limitations of the SSR. The SSR achieves either good dynamic range control or good color rendition, but not both. Jobson et al.12 observed a trade-off between accomplishing dynamic range control and achieving good color rendition that was modulated by the spatial constant of F(x,y) (i.e., the surround width): a large surround width providing the means to adjust the brightness of the image. This led them to utilize multiple widths of the surround at the same time, which is formalized as Display Formula

Ri(x,y)=s=1SWs{logIi(x,y)log[Fs(x,y)*Ii(x,y)]},(2)
with S being the number of different widths of surrounds and each center-surround operation weighted by Ws. MSR allows a way to balance between the dynamic range control and good color rendition by adjusting the surround widths and their respective weights per image application.

However, neither SSR nor MSR involved any reset or other global white- or grey-anchoring (e.g., global grayworld hypothesis) process, and thus output images of MSR Retinex tend to be desaturated. To overcome this issue, later studies incorporated an additional color restoration step following the Retinex application (MSR with color restoration; MSRCR).12,46 MSRCR overcomes the desaturation problem of MSR and maintains a good level of color constancy.

Multiscale NASA Retinex and Retina

The single-scale NASA Retinex conforms to the conventional view on the retinal lateral inhibition. However, the transition to the multiscale approach appears to be rather feasible given the latest neurophysiological evidence. As we briefly stated in Sec. 2.3, an increasing number of physiological studies report the existence of multiple scales of surround profiles (the “classic”28,30,31,32 and the “extraclassic”4042) in the retinal ganglion cell RF.36 This is comparable to the combined use of multiscale surrounds in the later version of the NASA Retinex.

The recent investigation by Yeonan-Kim and Bertalmío on the biological structure of the retinal processing evaluated the effect of the multiscale RF surrounds. As it turns out, the extraclassic (i.e., a larger spatial scale) surround produces a long-range spatial interaction, which the conventional concept of lateral inhibition (the “classic” RF) failed to explain. Some perceptual properties related to lightness induction4749 are likely engendered as the consequence of this long-range interaction. “Assimilation” phenomenon is an example. Assimilation refers to the case in which the lightness of an object tends to the luminance of the object’s surrounding [Fig. 1(a)], as opposed to the more typical case in which lightness shifts away from the surrounding luminance [“contrast” phenomenon; Fig. 1(b)]. While contrast is trivially attributed to the retinal lateral inhibition in the context of the classic RF, assimilation is influenced by the luminance of a remote surface that is too distant to be within the resolution of the classic ganglion cell RF. Yeonan-Kim and Bertalmío showed that the extraclassic surround produces this remote interaction.

Graphic Jump Location
Fig 1
F1 :

Lightness assimilation and contrast: (a) lightness assimilation phenomenon in which the gray bars of the same reflectance appear darker when embedded in the black bars (left) than in the white bars (right). (b) Lightness contrast phenomenon in which the gray patches of the same reflectance appear lighter inside a darker square (left) than in the lighter square (right).

Yeonan-Kim and Bertalmío’s study links the multiscale NASA Retinex to the physiological processing in the retina. The implication is that the large-scale surround in the NASA Retinex likely produces the long-range spatial effect similar to the extraclassic retinal RF. Conversely, that the multiscale structure similar to the NASA Retinex exists in the retinal circuitry implies that the retina may be capable of the kind of dynamic range control and color rendition that the NASA Retinex accomplishes.

Overall, we conclude that the spatial interaction operation of the designator and the NASA Retinexes are comparable to the retinal physiology.

Frankle and McCann7 patented a method to modify L&M Retinex3 to bypass using a path, which led to the development of the 2-D L&M Retinexes. As it turns out, these Retinexes share many interesting properties with a cortical vision model, the multiscale oriented DOG (ODOG) filtering model, incorporating the orientation and multiscale structure of the primary visual cortex, suggesting a link between the 2-D L&M Retinexes and cortical processing.

2-D implementation of L&M Retinex

The prototype L&M Retinex by Land and McCann3 introduced in 1971 took the turn toward 2-D implementation a few years later:

“In 1975, under the technical leadership of Jon Frankle, Polaroid purchased an I2S image processor (McCann 1983a, b; McCann, 2004). While the study of paths of Ratio-Product-Reset-Average model taught us about the nature of spatial interactions, it was hopelessly slow for processing real images. Influenced by the sampling techniques used by Stiehl et al. (1983), Frankle and McCann (1983) patented a multi-resolution algorithm that made it possible to process real images in real time.”6

Frankle and McCann’s7 patent detailed a new method removing the 1-D-path approach and adding multiscale operations. This led to the development of other 2-D L&M Retinex descendants,8,1315 all of which preserve the spatial interaction (ratio-product computation) and reset ideas of the prototype L&M Retinex, unlike the designator and NASA Retinexes.

Instead of proceeding along a path and selecting a pair of image points to compare, Frankle and McCann7 operated the ratio-product for the entire array of points in an image at once by computing the ratio between each point in an array to that in the spatially shifted version of the array iteratively while changing the shift direction at each iteration. For example, suppose I is the input image (calibrated such that the range of I codes the scene radiance in linear scale) and Rk is the ratio-product at k’th iteration. Then Display Formula

Rk(x,y)=I(x,y)I(xxk,yyk)·Rk1(xxk,yyk),(3)
where xk and yk specify the directions of the shift (the signs of xk and yk) and the pixel amounts of the shift (the absolutes values of x and y), respectively, at the k’th iteration. |xk| and |yk| start from the half-size of I (e.g., if the dimension of I is 512×512  pixels, |x1|=|y1|=256), which are then fixed for a user-designated number of iterations before decreasing by a factor of two for the next number of iterations. This strategy accomplishes the MSR computation: a large pixel shift at the beginning computes the ratio at a low-resolution, which changes to high-resolution with a decreasing amount of pixel shift.

Rk(x,y) is then reset Display Formula

Rk(x,y)={Rk(x,y),if  Rk(x,y)max(I)max(I),otherwise,(4)
and averaged Display Formula
Rk(x,y)=Rk(x,y)+Rk1(x,y)2,(5)
and the final output of the given iteration is assigned as the average between the current ratio-product and that of the previous.

2-D L&M Retinex by McCann

The update to Frankle and McCann’s7 2-D L&M Retinex was introduced by McCann8 (McCann 99 Retinex; see Ref. 9 for the patent documentation). While most of the algorithmic aspects remained comparable to the precedent, its major modification concerned the multiscale operation.

Frankle and McCann’s7 algorithm reduces the amount of pixel shift over iterations such that the ratio-product was computed in varying scales, from low- to high-resolution (Sec. 4.1). On the other hand, McCann 99 Retinex8 adopts a more straightforward approach to the resolution control: reducing the array size (pyramid processing). For example, the algorithm scales down the resolution of an input image, I, say, comprised of 512×512  pixels, to a 4×4  pixel array (call the resulting image array I4×4), each pixel containing the average of each 128×128 image block. The ratio-product at the k’th iteration is Display Formula

Rk4×4(x,y)=I4×4(x,y)I4×4(xxk,yyk)·Rk14×4(xxk,yyk).(6)

In this case, the maximum pixel shift is always one (i.e., xk[1,0,1] and yk[1,0,1]). The algorithm repeats the same operation for higher resolutions (R8×8, R16×16, and so on), until the resolution reaches that of the original image (R512×512). As in Frankle and McCann Retinex, Rk4×4(x,y) is then reset and averaged with Rk14×4(x,y) in the same way as in Eqs. (4) and (5).

Now let us rewrite Eq. (6) for an arbitrary scale, s, take the logarithm on both sides of the equation, and set x˜=xx and y˜=yy for convenience. Then we have Display Formula

logRks×s(x,y)=logIs×s(x,y)logIs×s(x˜,y˜)+logRk1s×s(x˜,y˜).(7)

Equation 7 shows that the ratio-product computes the difference (in log-scale) of the average of the reset product of one block of the image from that of the neighboring block. Note that, at each k, the shift direction changes: the differences are calculated for the horizontal direction, or orientation, with x˜k=±1 and y˜k=0, for the vertical orientation with x˜k=0 and y˜k=±1, and for diagonal orientations with the factorial combinations of x˜k=±1 and y˜k=±1. Again, Rks×s(x,y) is then reset Display Formula

Rks×s(x,y)={Rks×s(x,y),if  Rks×s(x,y)max(I)max(I),otherwise,(8)
and averaged Display Formula
Rks×s(x,y)=Rks×s(x,y)+Rk1s×s(x,y)2.(9)

This shows that McCann 99 Retinex computes the ratio-product over varying orientations at multiple scales of s, which is reminiscent of the orientation and spatial-frequency processing of the primary visual cortex (Sec. 2).

Vision Model on Cortical Processing: Multiscale ODOG Filtering Model

McCann 998 2-D L&M Retinex is analogous to a vision model published in the same year by Blakeslee and McCourt.50 This model was designed based on the physiological structure of the primary visual cortex that separately encodes varying orientations and spatial-frequency components of the scene. The idea was to process the input image, I, using multiple scales of ODOG kernels Display Formula

ODOGϕd,σs(x,y)=Gσs(x,y)Gϕd,σs(x,y),(10)
where Gσs(x,y)=Nex2+y22σs,x2 (N is a normalization constant) is a radially symmetric Gaussian kernel with the standard deviation, σs, where s is the nominal index of the size (scale) of the Gaussian. Gϕd,σs(x,y)=Ne(xϕd,x)22σs,x2+(yϕd,y)22σs,y2 is an elongated Gaussian with σs,y=2σs,x, and ϕd determines the d’th orientation of the elongation.

The output of the model, O, predicts the percept of IDisplay Formula

O(x,y)=1Dd=1Ds=1SWs{ODOGϕd,σs(x,y)*I(x,y)}s=1SWs{ODOGϕd,σs(x,y)*I(x,y)}x,y,(11)
which is obtained by filtering I with each ODOGϕd,σs, weighting the filtered image for each scale and summing over the scales for each orientation, normalizing for each orientation (d) these weighted sums by each of their own root-mean-square (RMS) contrasts [·x,y indicates RMS computation for all the points (x,y)], and then averaging across the orientation.

Throughout a series of studies,50,5155 Blakeslee and McCourt showed that the multiscale ODOG filtering model accounts for many lightness induction phenomena that were previously not explained. White’s illusion is a classic example (Fig. 2).50,56,57 In Fig. 2, the gray patches embedded in the vertical black and white bars have the same luminance, but the left patch appears lighter than the right patch. The direction of the lightness change is toward the luminance of the surround with which the patches have more contact (e.g., the left set of patches share more borders with the white than the black bars), and this phenomenon is thus classified as assimilation. This type of assimilation is not explained in terms of the retinal multiscale processing described in Sec. 3. On the other hand, the ODOG multiscale filtering model predicts White’s illusion.

Graphic Jump Location
Fig 2
F2 :

White’s Illusion: the two gray patches have the same reflectance, but the gray patch inside the black bar on the left appears lighter than the patch inside the white bar on the right.

McCourt, Blakeslee, and Cope58 recently extended the model application to Land and McCann’s3 Mondrian-like stimuli and demonstrated the operational similarity between the multiscale ODOG filtering model and the Retinex theory. While this demonstration was limited to lightness (rather than chromatic) induction by achromatic stimuli, applying the Retinex principle of channel independence (operating the above algorithm on each RGB channel) may expand the model feasibility to the color domain.

Comparing 2-D L&M Retinex to Multiscale ODOG Filtering Model

Both the McCann 99 2-D L&M Retinex and the multiscale ODOG filtering model involve the orientated multiscale processing structure that is evidenced in the architecture of the primary visual cortex. Of course, there are differences in terms of algorithmic details between them, but we leave more thorough mathematical analysis for future studies. Here, we focus on outlining the important similarities between them.

First of all, it is noticeable that the two algorithms sample the same type of information from the input image. The McCann 99 Retinex samples at a given resolution the difference of the average intensity at one region of the image from its neighborhoods aligned in a particular orientation [see the first two terms of Eq. (7)]. Filtering the input image with ODOGϕd,σs [Eq. (10)] realizes essentially the same sampling strategy except that it uses a Gaussian average rather than the blockwise mean.

Both of the algorithms then sum up the sampled information, albeit differently. 2-D L&M Retinex sums up the difference over the number of iterations across orientations first in the coarse scale and then proceeds to finer scales [note that this is not a linear summation due to the reset that follows the ratio-product computation; Eqs. (4) and (8)]. The multiscale ODOG filtering model linearly sums up across all the scales and then across the orientations [note that orientation-summation is not linear due to the normalization step; see Eq. (11)].

It is worth noting again that the 2-D L&M Retinex and the multiscale ODOG filtering model are models of human vision. While the L&M Retinex throughout history has aimed to predict color sensation,6,8,59 the multiscale ODOG filtering model was developed to predict brightness for a broad range of scene configurations, including the simple assimilation and contrast configurations (Figs. 1 and 2) and assimilation in more complicated patterns, as well as Adelson’s checkershadow illusion that was traditionally thought to involve illumination estimation.50,51,5355 Thus, the operational similarities between these two models58,60 imply some important common grounds in color and brightness processing that are worth investigating in the future.

The degree of similarity between the 2-D L&M Retinex and the multiscale ODOG filtering model in terms of information sampling structure and the performances suggests that the 2-D L&M Retinex may entail the cortex-like operation.

Varitional approaches for Retinex formulate Retinex as an optimization problem. They are largely subdivided into two groups, one that estimates the optimal contrast of an image21 and the other that estimates the illumination/reflectance of an image.1720,61 Both types are shown to be a simplified form of the Wilson–Cowan equation,20,62 a classic neural model for the cortical population responses,26 while their relationship to the visual system mechanisms is not clearly established. We mainly cover the documentations of the first group due to relevancy reasons, but the link between these Retinexes and the physiological model in our discussion holds for both groups.

Variational Model by Bertalmío et al.

Bertalmío et al.25 proposed a variational model for color and contrast enhancement based on other image processing techniques like the automatic color enhancement (ACE) algorithm of Rizzi et al.63 and the variational histogram equalization technique of Sapiro and Caselles.64 Although those earlier works did not consider any physiological evidence, later studies suggested a considerable algorithmic similarity between the variational model of Bertalmío et al.25 and a physiological model of cortical population responses, the Wilson–Cowan model65,66 (Sec. 5.2).

The algorithmic precedent of the variational model by Bertalmío et al.25 is the histogram equalization equation by Sapiro and Catelles64Display Formula

E(I)=2x[I(x)12]21ABxy|I(x)I(y)|,(12)
where x, y are pixels and A, B are the image dimensions, which is an energy functional to be minimized to obtain a flat histogram.

Bertalmío et al.25 modified Eq. (12) to conform to the perceptually based principles of the color enhancement technique, ACE, such as channel independence, spatial interaction, and grayworld hypothesis, and proposed a new energy functional Display Formula

E(I)=α2x[I(x)12]2γxyw(x,y)|I(x)I(y)|+β2x[I(x)I0(x)]2,(13)
where w(x,y) is a weighting function that scales the weight of y as its distance from x increases, I0 is the original image, and α, β, and γ are positive weights. Minimizing Eq. (13) locally enhances contrast (second term) and discounts the illuminant to preserve color constancy (first term), while preventing the image from departing too much from its original values (third term). Bertalmío et al.25 showed that Eq. (13) has a single minimum and that the image I minimizing Eq. (13) is a fixed point of ACE. This indicates that ACE is a numerical implementation of the gradient descent of Eq. (13).

Equation (13) has the gradient descent equation Display Formula

It(x)=α[I(x)12]+γyw(x,y)sgn[I(x)I(y)]β[I(x)I0(x)],(14)
and the minimum of the energy functional, Eq. (13), is obtained by updating It(x) in Eq. (14) iteratively from the initial values of I=I0 until It(x) reaches a steady state. The resulting I is the output of the model.

The explicit connection of the variational model by Bertalmío et al.25 to the Retinex theory was shown by Bertalmío et al.21 Here the authors proposed a 2-D implementation of the Retinex algorithm, named the KBR, which shares the essential elements of the 1-D-path based Milano Retinex algorithm:22,23Display Formula

L(x)=yAw(x,y)f[I(x)I(y)]sign+[I(y)I(x)]+yAw(x,y)sign[I(y)I(x)],(15)
where Display Formula
sign+(ξ)={1,if  sign+(ξ)>0,12,if  sign+(ξ)=0,0,if  sign+(ξ)<0,sign(ξ)=1sign+(ξ),
where A defines the image domain with symmetry assumption.21 While the Milano Retinex using paths suffers from image artifacts such as sensitivity to noise or appearance of halos, KBR does not produce such artifacts. Yet, KBR is unable to deal with overexposed images as is the Milano Retinex.

Bertalmío et al.21 proved that the limitation of KBR with overexposed images is imposed by the fact that there is no such energy that is minimized by the iterative application of the KBR algorithm. This led the authors to develop a modified version of KBR. As it turns out, this modified KBR equation is essentially the gradient descent, Eq. (14), of the energy functional in Eq. (13). In other words, the variational model of Bertalmío et al.25 is an iterative application of a modified version of KBR that improves on the handling of overexposed images.

Variational Retinexes to Wilson–Cowan Equation

Bertalmío and Cowan62 pointed out that the variational model by Bertalmío et al.25 resembles the Wilson–Cowan equation, a neural model that describes the population responses of the primary visual cortex using the PDE26Display Formula

a(r,ϕ,t)t=αa(r,ϕ,t)+μ0πR2ω(r,ϕr,ϕ)σ[a(r,ϕ,t)]drdϕ+h(r,ϕ,t),(16)
where the state a(r,ϕ,t) is the population of cells with cortical space coordinates rR2 and orientation preference ϕ[0,π)α, μ are coupling coefficients, h(r,ϕ,t) is the external input (visual stimuli), ω(r,ϕr,ϕ) is a kernel that decays with the differences |rr|, |ϕϕ|, and σ is a sigmoid function.

The similarity between the variational model by Bertalmío et al.25 and the neuroscience model becomes apparent with some generalization on Eq. (16). If we ignore the orientation ϕ and assume that the input h is constant in time, Eq. (16) takes the form Display Formula

a(r)t=αa(r)+μR2ω(rr)σ[a(r)]dr+h(r).(17)

Bertalmío and Cowan62 noted that the gradient descent of the energy functional of the variational model by Bertalmío et al.25 in Eq. (14) is nearly identical to the simplified, nonoriented version of the Wilson–Cowan equation in Eq. (17) if the absolute difference operation, |I(x)I(y)|, in the second term is replaced by σ[I(x)]. Note again that the variational model is intrinsically an iterative application of the Milano Retinex algorithm; this bridges the variational model to the physiological model structure developed in the neuroscience domain. At the same time, it is also implied that the Wilson–Cowan equation is the gradient descent of a certain energy.

It is worth noting that Zosso, Tran, and Osher20 also deduced the connection of the variational Retinexes to the Wilson–Cowan equation. Their work included an extensive review on the history of the family of variational Retinexes that formalize the Milano Retinex as a variational problem of illumination/reflectance estimation.1720,61 The authors then proposed a new form of nonlocal variational Retinex for reflectance estimation and showed that its estimate is equivalent to the steady-state form of Eq. (17).

Variational Retinexes in Relation to Other Retinex Families

The analysis of the variational Retinex models reveals potential ways to interpret diverse image processing problems (e.g., contrast and color enhancement, histogram modification, reflectance versus illumination parsing) in the context of the Retinex theory and the physiological mechanisms in the visual system. However, establishing an integrative perspective should not be hasty. Here, we address some precautions and prospects.

The algorithmic comparability between the variational Retinex to the nonoriented Wilson–Cowan equation [Eq. (17)] comes at the cost of the multiresolution processing and the orientation-dependent processing. It thus appears as if, at the end of the line, these Retinexes limit the functional structure of the spatial interaction to what is comparable to lateral inhibition, but in the conventional sense similar to Land’s designator idea.10 On the other hand, the multiresolution and the orientation-dependent schemes are important for predicting lightness.53,55,60,67

Bertalmío65 noted that the variational model [Eqs. (13) and (14)] does not reproduce the perceptual phenomena of assimilation and only generates a contrast effect. He subsequently proposed to modify the model, substituting the gray world hypothesis for a local mean and making the weight γ depend on local contrast, which yields the evolution equation Display Formula

It(x)=α[I(x)μ(x)]+γ(x)yw(x,y)sgn[I(x)I(y)]β[I(x)I0(x)].(18)

This gradient descent scheme maintains the same effect of contrast enhancement and attachment to I0, but now it tends to maintain the local mean within the vicinity defined by w(x,y) around the pixel (x,y) instead of gravitating toward the absolute global mean (i.e. gray). Equation (18) is shown to be able to reproduce assimilation effects comparable with human perception.47,68

There is an implicit link of Eq. (18) to the multiscale NASA Retinex, which may be worth noting. They both do not assume the global grayworld hypothesis and operate on the difference between the given pixel value with the local mean around it (“local” grayworld). In addition, Eq. (18) may produce a multiscale effect due to the two spatial terms, μ(x) and w(x,y), similar to the NASA Retinex. This requires future clarification.

Another issue is that the original Wilson–Cowan equation [Eq. (16)] entails the orientation processing. Given that the 2-D L&M Retinex and the multiscale ODOG filtering model, with “oriented” multiscale processing, are capable of predicting human lightness data in diverse scenarios (Sec. 4.4), it may be of interest to consider a kind of “multiscale” Wilson–Cowan equation rather than the simplified version. This may provide a more formal analytic means to assess the relationship between the variational Retinexes and other families of Retinexes, as well as the physiological mechanisms.

Overall, we conclude that the variational Retinexes, at least in the current forms, are retina-like as designator and NASA Retinexes in their spatial processing structures. All these Retinexes disregard reset operation unlike L&M Retinex. Interestingly, the way performances of these retina-like Retinex algorithms are evaluated is through visual inspection or preference measures of visibly acceptable or pleasing images. This is different from the original intention of the L&M Retinex, which was to predict the color sensation of human observers. McCann69 pointed out, however, that visual inspection cannot provide means to evaluate a vision model since in this scheme the output of “a vision model” would pass through the visual system again before evaluation, which is problematic. Alternatively, the retina-like Retinexes need to be considered as vision-inspired image processing algorithms rather than a vision model. At least, the goal of the retina-like algorithms is to improve viewers’ experiences, which is distinguished from the original L&M Retinex.

So far we reviewed various Retinex algorithms in their relation to the physiological mechanisms in the retina and the primary visual cortex. Our intention was to suggest an initial framework to which various Retinex algorithms are compared. The effort has yet to quantitatively or mathematically evaluate the Retinex variations regarding their computational time, performance, predictability to human perception, coherence to the physiological system, and compatibility with other image processing tools. Linking the Retinex theory to the physiological mechanisms can also lead to a better understanding of the functional implications of the physiological structure.

The Retinex theory appears to encompass other image processing problems. Many of these problems revolve around the issues of chromatic and achromatic contrast processing that codes spatial intensity differences in the scene. As was recognized by Barlow from about half a century ago70,71 (see Ref. 72 for a review), minimizing the spatial redundancies is the fundamental strategy of biological vision.73,74 Understanding this spatial interaction strategy both from a biological and a technical perspective may thus provide further insights on the Retinex theory and assist the development of relevant image processing techniques.

This work was supported by the European Research Council, Starting Grant Ref. 306337, by the Spanish government and FEDER Fund, grant Ref. TIN2015-71537-P(MINECO/FEDER,UE), and by the Icrea Academia Award. The authors declare that no conflicting interests exist.

Land  E., “The Retinex theory of color vision,” Sci. Am.. 237, (6 ), 108 –128 (1977). 0036-8733 CrossRef
Land  E. H., “The Retinex,” Am. Sci.. 52, (2 ), 247 –264 (1964). 0003-0996 
Land  E., and McCann  J., “Lightness and Retinex theory,” J. Opt. Soc. Am.. 61, , 1 –11 (1971). 0030-3941 CrossRef
McCann  J. J., , Land  E. H., and Tatnall  S. M., “A technique for comparing human visual responses with a mathematical model for lightness,” Optom. Vision Sci.. 47, (11 ), 845 –855 (1970).CrossRef
McCann  J. J., , McKee  S. P., and Taylor  T. H., “Quantitative studies in Retinex theory a comparison between theoretical predictions and observer responses to the color Mondrian experiments,” Vision Res.. 16, (5 ), 445 –458 (1976). 0042-6989 CrossRef
McCann  J. J., and Rizzi  A., “The art and science of HDR imaging,” in Retinex Algorithms. , Vol. 26, ,  John Wiley & Sons ,  Chichester, UK  (2011).
Frankle  J., and McCann  J., “Method and apparatus of lightness imaging,” US 4384336 A (1983).
McCann  J., “Lessons learned from Mondrians applied to real images and color gamuts,” in  Color and Imaging Conf. , pp. 1 –8 (1999).
Wray  W. R., “Method and apparatus for image processing with field portions,” US Patent 4, 750, 211 (1988).
Land  E., “An alternative technique for the computation of the designator in the Retinex theory of color vision,” Proc. Natl. Acad. Sci. U. S. A.. 83, , 3078 –3080 (1986).CrossRef
Jobson  D. J., , Rahman  Z. U., and Woodell  G. A., “Properties and performance of a center/surround Retinex,” IEEE Trans. Image Process.. 6, , 451 –462 (1997). 1057-7149 CrossRef
Jobson  D. J., , Rahman  Z. U., and Woodell  G. A., “A multiscale Retinex for bridging the gap between color images and the human observation of scenes,” IEEE Trans. Image Process.. 6, , 965 –976 (1997). 1057-7149 CrossRef
McCann  J. J., “Using color constancy to advantage in color gamut calculations,” in  Proc. IS and T PICS Conf. , pp. 169 –176 (2000).
Sobol  R., and McCann  J., “Method for variable contrast mapping of digital images,” US Patent App. 09/803, 469 (2001).
Sobol  R., “Improving the Retinex algorithm for rendering wide dynamic range photographs,” J. Electron. Imaging. 13, (1 ), 65 –74 (2004). 1017-9909 CrossRef
Kotera  H., and Fujita  M., “Appearance improvement of color image by adaptive scale-gain Retinex model,” in  Color and Imaging Conf. , pp. 166 –171 (2002).
Kimmel  R.  et al., “A variational framework for Retinex,” Int. J. Comput. Vision. 52, (1 ), 7 –23 (2003). 0920-5691 CrossRef
Ma  W., and Osher  S., “A TV Bergman iterative model of Retinex theory,” UCLA CAM Report, pp. 10 –13 (2010).
Ng  M. K., and Wang  W., “A total variation model for Retinex,” SIAM J. Imag. Sci.. 4, (1 ), 345 –365 (2011).CrossRef
Zosso  D., , Tran  G., and Osher  S. J., “Non-local Retinex—a unifying framework and beyond,” SIAM J. Imag. Sci.. 8, (2 ), 787 –826 (2015).CrossRef
Bertalmío  M., , Caselles  V., and Provenzi  E., “Issues about Retinex theory and contrast enhancement,” Int. J. Comput. Vision. 83, (1 ), 101 –119 (2009). 0920-5691 CrossRef
Marini  D., and Rizzi  A., “A computational approach to color adaptation effects,” Image Vision Comput.. 18, (13 ), 1005 –1014 (2000). 0262-8856 CrossRef
Provenzi  E.  et al., “Mathematical definition and analysis of the Retinex algorithm,” J. Opt. Soc. Am. A. 22, , 2613 –2621 (2005). 0740-3232 CrossRef
Land  E. H., “Recent advances in Retinex theory and some implications for cortical computations: color vision and the natural image,” Proc. Natl. Acad. Sci. U. S. A.. 80, (16 ), 5163 –5169 (1983).CrossRef
Bertalmío  M.  et al., “Perceptual color correction through variational techniques,” IEEE Trans. Image Process.. 16, , 1058 –1072 (2007). 1057-7149 CrossRef
Bressloff  P.  et al., “What geometric visual hallucinations tell us about the visual cortex,” Neural Comput.. 14, (3 ), 473 –491 (2002). 0899-7667 CrossRef
Thoreson  W. B., and Mangel  S. C., “Lateral interactions in the outer retina,” Prog. Retinal Eye Res.. 31, (5 ), 407 –441 (2012). 1350-9462 CrossRef
Enroth-Cugell  C., and Robson  J. G., “The contrast sensitivity of retinal ganglion cells of the cat,” J. Physiol.. 187, , 517 –552 (1966). 0022-3751 CrossRef
Kuffler  S. W., “Discharge patterns and functional organization of mammalian retina,” J. Neurophysiol.. 16, (1 ), 37 –68 (1953). 0022-3077 
Ratliff  F., “Contour and contrast,” Proc. Am. Philos. Soc.. 115, (2 ), 150 –163 (1971).
Rodieck  R. W., and Stone  J., “Analysis of receptive fields of cat retinal ganglion cells,” J. Neurophysiol.. 28, (5 ), 833 –849 (1965). 0022-3077 
Shapley  R., , Enroth-Cugell  C., “Visual adaptation and retinal gain controls,” in Progress in Retinal Research. , , Osborne  N., and Chader  G., Eds., Vol. 3, , pp. 263 –346,  Pergamon ,  Oxford  (1984).
Hubel  D. H., and Wiesel  T. N., “Uniformity of monkey striate cortex: a parallel relationship between field size, scatter, and magnification factor,” J. Comp. Neurol.. 158, (3 ), 295 –305 (1974). 0021-9967 CrossRef
Hubel  D. H., and Wiesel  T. N., “Ferrier lecture: functional architecture of macaque monkey visual cortex,” Proc. R. Soc. London B Biol. Sci.. 198, (1130 ), 1 –59 (1977).CrossRef
Blakemore  C., and Campbell  F., “On the existence of neurones in the human visual system selectively sensitive to the orientation and size of retinal images,” J. Physiol.. 203, (1 ), 237 –260 (1969).CrossRef
Yeonan-Kim  J., and Bertalmío  M., “Retinal lateral inhibition provides the biological basis of long-range spatial induction,” PloS ONE. 11, (12 ), e0168963  (2016). 1932-6203 CrossRef
Packer  O. S., and Dacey  D. M., “Receptive field structure of H1 horizontal cells in macaque monkey retina,” J. Vision. 2, (4 ), 1 –1 (2002). 1534-7362 CrossRef
Manookin  M. B.  et al., “Distinctive receptive field and physiological properties of a wide-field amacrine cell in the macaque monkey retina,” J. Neurophysiol.. 114, (3 ), 1606 –1616 (2015). 0022-3077 CrossRef
Kolb  H., “Amacrine cells of the mammalian retina: neurocircuitry and functional roles,” Eye. 11, , 904 –923 (1997).CrossRef
Passaglia  C. L., , Enroth-Cugell  C., and Troy  J. B., “Effects of remote stimulation on the mean firing rate of cat retinal ganglion cells,” J. Neurosci.. 21, (15 ), 5794 –5803 (2001). 0270-6474 
Solomon  S. G., , Lee  B. B., and Sun  H., “Suppressive surrounds and contrast gain in magnocellular-pathway retinal ganglion cells of macaque,” J. Neurosci.. 26, (34 ), 8715 –8726 (2006). 0270-6474 CrossRef
Kaplan  E., and Benardete  E., “The dynamics of primate retinal ganglion cells,” Prog. Brain Res.. 134, , 17 –34 (2001). 0079-6123 CrossRef
Horn  B. K., “Determining lightness from an image,” Comput. Graphics Image Process.. 3, (4 ), 277 –299 (1974). 0146-664X CrossRef
Hurlbert  A., “Formal connections between lightness algorithms,” J. Opt. Soc. Am. A. 3, (10 ), 1684 –1693 (1986).CrossRef
Moore  A., , Allman  J., and Goodman  R. M., “A real-time neural system for color constancy,” IEEE Trans. Neural Networks. 2, (2 ), 237 –247 (1991). 1045-9227 CrossRef
Rahman  Z.-U., , Jobson  D. J., and Woodell  G. A., “Retinex processing for automatic image enhancement,” J. Electron. Imaging. 13, , 100 –110 (2004). 1017-9909 CrossRef
Helson  H., “Studies of anomalous contrast and assimilation,” J. Opt. Soc. Am.. 53, , 179 –184 (1963). 0030-3941 CrossRef
Reid  R. C., and Shapley  R., “Brightness induction by local contrast and the spatial dependence of assimilation,” Vision Res.. 28, (1 ), 115 –132 (1988). 0042-6989 CrossRef
Rudd  M. E., and Zemach  I. K., “Quantitative properties of achromatic color induction: an edge integration analysis,” Vision Res.. 44, , 971 –981 (2004). 0042-6989 CrossRef
Blakeslee  B., and McCourt  M. E., “A multiscale spatial filtering account of the white effect, simultaneous brightness contrast and grating induction,” Vision Res.. 39, , 4361 –4377 (1999). 0042-6989 CrossRef
Blakeslee  B., and McCourt  M. E., “A unified theory of brightness contrast and assimilation incorporating oriented multiscale spatial filtering and contrast normalization,” Vision Res.. 44, , 2483 –2503 (2004). 0042-6989 CrossRef
Blakeslee  B., and McCourt  M. E., “Oriented multiscale spatial filtering and contrast normalization: a parsimonious model of brightness induction in a continuum of stimuli including White, Howe and simultaneous brightness contrast,” Vision Res.. 45, , 607 –615 (2005). 0042-6989 CrossRef
Blakeslee  B., and McCourt  M. E., “When is spatial filtering enough? Investigation of brightness and lightness perception in stimuli containing a visible illumination component,” Vision Res.. 60, , 40 –50 (2012). 0042-6989 CrossRef
Blakeslee  B., and McCourt  M. E., “Brightness induction magnitude declines with increasing distance from the inducing field edge,” Vision Res.. 78, , 39 –45 (2013). 0042-6989 CrossRef
Blakeslee  B., and McCourt  M. E., “What visual illusions tell us about underlying neural mechanisms and observer strategies for tackling the inverse problem of achromatic perception,” Front. Hum. Neurosci.. 9, , 205  (2015).CrossRef
White  M., “A new effect of pattern on perceived lightness,” Perception. 8, , 413 –416 (1979). 0301-0066 CrossRef
White  M., “The effect of the nature of the surround on the perceived lightness of grey bars within square-wave test gratings,” Perception. 10, , 215 –230 (1981). 0301-0066 CrossRef
McCourt  M. E., , Blakeslee  B., and Cope  D., “The oriented difference-of-Gaussians model of brightness perception,” Electron. Imaging. 2016, , 1 –9 (2016).CrossRef
McCann  J. J., “Capturing a black cat in shade: past and present of Retinex color appearance models,” J. Electron. Imaging. 13, (1 ), 36 –47 (2004). 1017-9909 CrossRef
McCann  J., “Calculating lightnesses in a single depth plane,” J. Electron. Imaging. 10, (1 ), 110 –122 (2001). 1017-9909 CrossRef
Morel  J. M., , Petro  A. B., and Sbert  C., “A pde formalization of Retinex theory,” IEEE Trans. Image Process.. 19, (11 ), 2825 –2837 (2010). 1057-7149 CrossRef
Bertalmío  M., and Cowan  J., “Implementing the Retinex algorithm with Wilson-Cowan equations,” J. Physiol. Paris. 103, , 69 –72 (2009).CrossRef
Rizzi  A., , Gatta  C., and Marini  D., “A new algorithm for unsupervised global and local color correction,” Pattern Recognit. Lett.. 24, , 1663 –1677 (2003). 0167-8655 CrossRef
Sapiro  G., and Caselles  V., “Histogram modification via differential equations,” J. Differ. Equ.. 135, , 238 –268 (1997). 0022-0396 CrossRef
Bertalmío  M., “From image processing to computational neuroscience: a neural model based on histogram equalization,” Front. Comput. Neurosci.. 8, , 71  (2014).CrossRef
Bertalmío  M., “Modeling lightness induction with Wilson-Cowan equations,” J. Vision. 14, (15 ), 56 –56 (2014). 1534-7362 CrossRef
McCann  J. J., “Image processing analysis of traditional gestalt vision experiments,” Proc. SPIE. 4421, , 375  (2002).CrossRef
Rudd  M., “How attention and contrast gain control interact to regulate lightness contrast and assimilation: a computational neural model,” J. Vision. 10, , 40  (2010). 1534-7362 CrossRef
McCann  J. J., “Retinex at 50: color theory and spatial algorithms, a review,” J. Electron. Imaging. 26, (3 ), 031204  (2017). 1017-9909 CrossRef
Barlow  H. B., “Possible principles underlying the transformations of sensory messages,” in Sensory Communication. , and Rosenblith  W. A., Ed., pp. 217 –234,  MIT Press ,  Cambridge  (1961).
Barlow  H. B., “Three points about lateral inhibition,” in Sensory Communication. , and Rosenblith  W. A., Ed., 782 –786,  MIT Press ,  Cambridge  (1961).
Olshausen  B., and Field  D., “Vision and the coding of natural images,” Am. Sci.. 88, (3 ), 238 –245 (2000). 0003-0996 CrossRef
Atick  J., “Could information theory provide an ecological theory of sensory processing?” Network Comput. Neural Syst.. 3, (2 ), 213 –251 (1992).CrossRef
Dan  Y., , Atick  J. J., and Reid  R. C., “Efficient coding of natural scenes in the lateral geniculate nucleus: experimental test of a computational theory,” J. Neurosci.. 16, (10 ), 3351 –3362 (1996).

Jihyun Yeonan-Kim completed her PhD in psychological sciences and her MS degree in electrical engineering at Purdue University in 2013. She is currently a postdoctoral researcher at the University Pompeu Fabra, Spain. She received traditional training in psychophysics and neuroscience and now develops the biophysical models of retinal and early cortical circuitry for both theoretic and applicational purposes. She paints and photographs to explore human perception and imaging technologies.

Marcelo Bertalmío received his PhD in electrical and computer engineering from the University of Minnesota in 2001. He is an associate professor at the University Pompeu Fabra in Barcelona, Spain. His interests are image processing and computer vision for digital cinema applications, although he prefers the (analog) films of Luis Buñuel and Luis García Berlanga.

© The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Citation

Jihyun Yeonan-Kim and Marcelo Bertalmío
"Analysis of retinal and cortical components of Retinex algorithms", J. Electron. Imaging. 26(3), 031208 (May 17, 2017). ; http://dx.doi.org/10.1117/1.JEI.26.3.031208


Figures

Graphic Jump Location
Fig 1
F1 :

Lightness assimilation and contrast: (a) lightness assimilation phenomenon in which the gray bars of the same reflectance appear darker when embedded in the black bars (left) than in the white bars (right). (b) Lightness contrast phenomenon in which the gray patches of the same reflectance appear lighter inside a darker square (left) than in the lighter square (right).

Graphic Jump Location
Fig 2
F2 :

White’s Illusion: the two gray patches have the same reflectance, but the gray patch inside the black bar on the left appears lighter than the patch inside the white bar on the right.

Tables

References

Land  E., “The Retinex theory of color vision,” Sci. Am.. 237, (6 ), 108 –128 (1977). 0036-8733 CrossRef
Land  E. H., “The Retinex,” Am. Sci.. 52, (2 ), 247 –264 (1964). 0003-0996 
Land  E., and McCann  J., “Lightness and Retinex theory,” J. Opt. Soc. Am.. 61, , 1 –11 (1971). 0030-3941 CrossRef
McCann  J. J., , Land  E. H., and Tatnall  S. M., “A technique for comparing human visual responses with a mathematical model for lightness,” Optom. Vision Sci.. 47, (11 ), 845 –855 (1970).CrossRef
McCann  J. J., , McKee  S. P., and Taylor  T. H., “Quantitative studies in Retinex theory a comparison between theoretical predictions and observer responses to the color Mondrian experiments,” Vision Res.. 16, (5 ), 445 –458 (1976). 0042-6989 CrossRef
McCann  J. J., and Rizzi  A., “The art and science of HDR imaging,” in Retinex Algorithms. , Vol. 26, ,  John Wiley & Sons ,  Chichester, UK  (2011).
Frankle  J., and McCann  J., “Method and apparatus of lightness imaging,” US 4384336 A (1983).
McCann  J., “Lessons learned from Mondrians applied to real images and color gamuts,” in  Color and Imaging Conf. , pp. 1 –8 (1999).
Wray  W. R., “Method and apparatus for image processing with field portions,” US Patent 4, 750, 211 (1988).
Land  E., “An alternative technique for the computation of the designator in the Retinex theory of color vision,” Proc. Natl. Acad. Sci. U. S. A.. 83, , 3078 –3080 (1986).CrossRef
Jobson  D. J., , Rahman  Z. U., and Woodell  G. A., “Properties and performance of a center/surround Retinex,” IEEE Trans. Image Process.. 6, , 451 –462 (1997). 1057-7149 CrossRef
Jobson  D. J., , Rahman  Z. U., and Woodell  G. A., “A multiscale Retinex for bridging the gap between color images and the human observation of scenes,” IEEE Trans. Image Process.. 6, , 965 –976 (1997). 1057-7149 CrossRef
McCann  J. J., “Using color constancy to advantage in color gamut calculations,” in  Proc. IS and T PICS Conf. , pp. 169 –176 (2000).
Sobol  R., and McCann  J., “Method for variable contrast mapping of digital images,” US Patent App. 09/803, 469 (2001).
Sobol  R., “Improving the Retinex algorithm for rendering wide dynamic range photographs,” J. Electron. Imaging. 13, (1 ), 65 –74 (2004). 1017-9909 CrossRef
Kotera  H., and Fujita  M., “Appearance improvement of color image by adaptive scale-gain Retinex model,” in  Color and Imaging Conf. , pp. 166 –171 (2002).
Kimmel  R.  et al., “A variational framework for Retinex,” Int. J. Comput. Vision. 52, (1 ), 7 –23 (2003). 0920-5691 CrossRef
Ma  W., and Osher  S., “A TV Bergman iterative model of Retinex theory,” UCLA CAM Report, pp. 10 –13 (2010).
Ng  M. K., and Wang  W., “A total variation model for Retinex,” SIAM J. Imag. Sci.. 4, (1 ), 345 –365 (2011).CrossRef
Zosso  D., , Tran  G., and Osher  S. J., “Non-local Retinex—a unifying framework and beyond,” SIAM J. Imag. Sci.. 8, (2 ), 787 –826 (2015).CrossRef
Bertalmío  M., , Caselles  V., and Provenzi  E., “Issues about Retinex theory and contrast enhancement,” Int. J. Comput. Vision. 83, (1 ), 101 –119 (2009). 0920-5691 CrossRef
Marini  D., and Rizzi  A., “A computational approach to color adaptation effects,” Image Vision Comput.. 18, (13 ), 1005 –1014 (2000). 0262-8856 CrossRef
Provenzi  E.  et al., “Mathematical definition and analysis of the Retinex algorithm,” J. Opt. Soc. Am. A. 22, , 2613 –2621 (2005). 0740-3232 CrossRef
Land  E. H., “Recent advances in Retinex theory and some implications for cortical computations: color vision and the natural image,” Proc. Natl. Acad. Sci. U. S. A.. 80, (16 ), 5163 –5169 (1983).CrossRef
Bertalmío  M.  et al., “Perceptual color correction through variational techniques,” IEEE Trans. Image Process.. 16, , 1058 –1072 (2007). 1057-7149 CrossRef
Bressloff  P.  et al., “What geometric visual hallucinations tell us about the visual cortex,” Neural Comput.. 14, (3 ), 473 –491 (2002). 0899-7667 CrossRef
Thoreson  W. B., and Mangel  S. C., “Lateral interactions in the outer retina,” Prog. Retinal Eye Res.. 31, (5 ), 407 –441 (2012). 1350-9462 CrossRef
Enroth-Cugell  C., and Robson  J. G., “The contrast sensitivity of retinal ganglion cells of the cat,” J. Physiol.. 187, , 517 –552 (1966). 0022-3751 CrossRef
Kuffler  S. W., “Discharge patterns and functional organization of mammalian retina,” J. Neurophysiol.. 16, (1 ), 37 –68 (1953). 0022-3077 
Ratliff  F., “Contour and contrast,” Proc. Am. Philos. Soc.. 115, (2 ), 150 –163 (1971).
Rodieck  R. W., and Stone  J., “Analysis of receptive fields of cat retinal ganglion cells,” J. Neurophysiol.. 28, (5 ), 833 –849 (1965). 0022-3077 
Shapley  R., , Enroth-Cugell  C., “Visual adaptation and retinal gain controls,” in Progress in Retinal Research. , , Osborne  N., and Chader  G., Eds., Vol. 3, , pp. 263 –346,  Pergamon ,  Oxford  (1984).
Hubel  D. H., and Wiesel  T. N., “Uniformity of monkey striate cortex: a parallel relationship between field size, scatter, and magnification factor,” J. Comp. Neurol.. 158, (3 ), 295 –305 (1974). 0021-9967 CrossRef
Hubel  D. H., and Wiesel  T. N., “Ferrier lecture: functional architecture of macaque monkey visual cortex,” Proc. R. Soc. London B Biol. Sci.. 198, (1130 ), 1 –59 (1977).CrossRef
Blakemore  C., and Campbell  F., “On the existence of neurones in the human visual system selectively sensitive to the orientation and size of retinal images,” J. Physiol.. 203, (1 ), 237 –260 (1969).CrossRef
Yeonan-Kim  J., and Bertalmío  M., “Retinal lateral inhibition provides the biological basis of long-range spatial induction,” PloS ONE. 11, (12 ), e0168963  (2016). 1932-6203 CrossRef
Packer  O. S., and Dacey  D. M., “Receptive field structure of H1 horizontal cells in macaque monkey retina,” J. Vision. 2, (4 ), 1 –1 (2002). 1534-7362 CrossRef
Manookin  M. B.  et al., “Distinctive receptive field and physiological properties of a wide-field amacrine cell in the macaque monkey retina,” J. Neurophysiol.. 114, (3 ), 1606 –1616 (2015). 0022-3077 CrossRef
Kolb  H., “Amacrine cells of the mammalian retina: neurocircuitry and functional roles,” Eye. 11, , 904 –923 (1997).CrossRef
Passaglia  C. L., , Enroth-Cugell  C., and Troy  J. B., “Effects of remote stimulation on the mean firing rate of cat retinal ganglion cells,” J. Neurosci.. 21, (15 ), 5794 –5803 (2001). 0270-6474 
Solomon  S. G., , Lee  B. B., and Sun  H., “Suppressive surrounds and contrast gain in magnocellular-pathway retinal ganglion cells of macaque,” J. Neurosci.. 26, (34 ), 8715 –8726 (2006). 0270-6474 CrossRef
Kaplan  E., and Benardete  E., “The dynamics of primate retinal ganglion cells,” Prog. Brain Res.. 134, , 17 –34 (2001). 0079-6123 CrossRef
Horn  B. K., “Determining lightness from an image,” Comput. Graphics Image Process.. 3, (4 ), 277 –299 (1974). 0146-664X CrossRef
Hurlbert  A., “Formal connections between lightness algorithms,” J. Opt. Soc. Am. A. 3, (10 ), 1684 –1693 (1986).CrossRef
Moore  A., , Allman  J., and Goodman  R. M., “A real-time neural system for color constancy,” IEEE Trans. Neural Networks. 2, (2 ), 237 –247 (1991). 1045-9227 CrossRef
Rahman  Z.-U., , Jobson  D. J., and Woodell  G. A., “Retinex processing for automatic image enhancement,” J. Electron. Imaging. 13, , 100 –110 (2004). 1017-9909 CrossRef
Helson  H., “Studies of anomalous contrast and assimilation,” J. Opt. Soc. Am.. 53, , 179 –184 (1963). 0030-3941 CrossRef
Reid  R. C., and Shapley  R., “Brightness induction by local contrast and the spatial dependence of assimilation,” Vision Res.. 28, (1 ), 115 –132 (1988). 0042-6989 CrossRef
Rudd  M. E., and Zemach  I. K., “Quantitative properties of achromatic color induction: an edge integration analysis,” Vision Res.. 44, , 971 –981 (2004). 0042-6989 CrossRef
Blakeslee  B., and McCourt  M. E., “A multiscale spatial filtering account of the white effect, simultaneous brightness contrast and grating induction,” Vision Res.. 39, , 4361 –4377 (1999). 0042-6989 CrossRef
Blakeslee  B., and McCourt  M. E., “A unified theory of brightness contrast and assimilation incorporating oriented multiscale spatial filtering and contrast normalization,” Vision Res.. 44, , 2483 –2503 (2004). 0042-6989 CrossRef
Blakeslee  B., and McCourt  M. E., “Oriented multiscale spatial filtering and contrast normalization: a parsimonious model of brightness induction in a continuum of stimuli including White, Howe and simultaneous brightness contrast,” Vision Res.. 45, , 607 –615 (2005). 0042-6989 CrossRef
Blakeslee  B., and McCourt  M. E., “When is spatial filtering enough? Investigation of brightness and lightness perception in stimuli containing a visible illumination component,” Vision Res.. 60, , 40 –50 (2012). 0042-6989 CrossRef
Blakeslee  B., and McCourt  M. E., “Brightness induction magnitude declines with increasing distance from the inducing field edge,” Vision Res.. 78, , 39 –45 (2013). 0042-6989 CrossRef
Blakeslee  B., and McCourt  M. E., “What visual illusions tell us about underlying neural mechanisms and observer strategies for tackling the inverse problem of achromatic perception,” Front. Hum. Neurosci.. 9, , 205  (2015).CrossRef
White  M., “A new effect of pattern on perceived lightness,” Perception. 8, , 413 –416 (1979). 0301-0066 CrossRef
White  M., “The effect of the nature of the surround on the perceived lightness of grey bars within square-wave test gratings,” Perception. 10, , 215 –230 (1981). 0301-0066 CrossRef
McCourt  M. E., , Blakeslee  B., and Cope  D., “The oriented difference-of-Gaussians model of brightness perception,” Electron. Imaging. 2016, , 1 –9 (2016).CrossRef
McCann  J. J., “Capturing a black cat in shade: past and present of Retinex color appearance models,” J. Electron. Imaging. 13, (1 ), 36 –47 (2004). 1017-9909 CrossRef
McCann  J., “Calculating lightnesses in a single depth plane,” J. Electron. Imaging. 10, (1 ), 110 –122 (2001). 1017-9909 CrossRef
Morel  J. M., , Petro  A. B., and Sbert  C., “A pde formalization of Retinex theory,” IEEE Trans. Image Process.. 19, (11 ), 2825 –2837 (2010). 1057-7149 CrossRef
Bertalmío  M., and Cowan  J., “Implementing the Retinex algorithm with Wilson-Cowan equations,” J. Physiol. Paris. 103, , 69 –72 (2009).CrossRef
Rizzi  A., , Gatta  C., and Marini  D., “A new algorithm for unsupervised global and local color correction,” Pattern Recognit. Lett.. 24, , 1663 –1677 (2003). 0167-8655 CrossRef
Sapiro  G., and Caselles  V., “Histogram modification via differential equations,” J. Differ. Equ.. 135, , 238 –268 (1997). 0022-0396 CrossRef
Bertalmío  M., “From image processing to computational neuroscience: a neural model based on histogram equalization,” Front. Comput. Neurosci.. 8, , 71  (2014).CrossRef
Bertalmío  M., “Modeling lightness induction with Wilson-Cowan equations,” J. Vision. 14, (15 ), 56 –56 (2014). 1534-7362 CrossRef
McCann  J. J., “Image processing analysis of traditional gestalt vision experiments,” Proc. SPIE. 4421, , 375  (2002).CrossRef
Rudd  M., “How attention and contrast gain control interact to regulate lightness contrast and assimilation: a computational neural model,” J. Vision. 10, , 40  (2010). 1534-7362 CrossRef
McCann  J. J., “Retinex at 50: color theory and spatial algorithms, a review,” J. Electron. Imaging. 26, (3 ), 031204  (2017). 1017-9909 CrossRef
Barlow  H. B., “Possible principles underlying the transformations of sensory messages,” in Sensory Communication. , and Rosenblith  W. A., Ed., pp. 217 –234,  MIT Press ,  Cambridge  (1961).
Barlow  H. B., “Three points about lateral inhibition,” in Sensory Communication. , and Rosenblith  W. A., Ed., 782 –786,  MIT Press ,  Cambridge  (1961).
Olshausen  B., and Field  D., “Vision and the coding of natural images,” Am. Sci.. 88, (3 ), 238 –245 (2000). 0003-0996 CrossRef
Atick  J., “Could information theory provide an ecological theory of sensory processing?” Network Comput. Neural Syst.. 3, (2 ), 213 –251 (1992).CrossRef
Dan  Y., , Atick  J. J., and Reid  R. C., “Efficient coding of natural scenes in the lateral geniculate nucleus: experimental test of a computational theory,” J. Neurosci.. 16, (10 ), 3351 –3362 (1996).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Related Book Chapters

Topic Collections

PubMed Articles
Advertisement
  • Don't have an account?
  • Subscribe to the SPIE Digital Library
  • Create a FREE account to sign up for Digital Library content alerts and gain access to institutional subscriptions remotely.
Access This Article
Sign in or Create a personal account to Buy this article ($20 for members, $25 for non-members).
Access This Proceeding
Sign in or Create a personal account to Buy this article ($15 for members, $18 for non-members).
Access This Chapter

Access to SPIE eBooks is limited to subscribing institutions and is not available as part of a personal subscription. Print or electronic versions of individual SPIE books may be purchased via SPIE.org.