Regular Articles

Color correction pipeline optimization for digital cameras

[+] Author Affiliations
Simone Bianco

University of Milano-Bicocca, Department of Informatics, Systems and Communication, Viale Sarca 336, Building U14, 20126 Milano, Italy

Arcangelo R. Bruna

STMicroelectronics, AST Advanced System Technology—Catania Lab, Catania, Italy

Filippo Naccari

STMicroelectronics, AST Advanced System Technology—Catania Lab, Catania, Italy

Raimondo Schettini

University of Milano-Bicocca, Department of Informatics, Systems and Communication, Viale Sarca 336, Building U14, 20126 Milano, Italy

J. Electron. Imaging. 22(2), 023014 (May 14, 2013). doi:10.1117/1.JEI.22.2.023014
History: Received August 28, 2012; Revised March 13, 2013; Accepted April 17, 2013
Text Size: A A A

Open Access Open Access

Abstract.  The processing pipeline of a digital camera converts the RAW image acquired by the sensor to a representation of the original scene that should be as faithful as possible. There are mainly two modules responsible for the color-rendering accuracy of a digital camera: the former is the illuminant estimation and correction module, and the latter is the color matrix transformation aimed to adapt the color response of the sensor to a standard color space. These two modules together form what may be called the color correction pipeline. We design and test new color correction pipelines that exploit different illuminant estimation and correction algorithms that are tuned and automatically selected on the basis of the image content. Since the illuminant estimation is an ill-posed problem, illuminant correction is not error-free. An adaptive color matrix transformation module is optimized, taking into account the behavior of the first module in order to alleviate the amplification of color errors. The proposed pipelines are tested on a publicly available dataset of RAW images. Experimental results show that exploiting the cross-talks between the modules of the pipeline can lead to a higher color-rendition accuracy.

Figures in this Article

The color rendering accuracy of a digital imaging acquisition device is a key factor to the overall perceived image quality.13 There are mainly two modules responsible for the color rendering accuracy in a digital camera: the former is the illuminant estimation and correction module, the latter is the color matrix transformation. These two modules together form what may be called the color correction pipeline.

The first stage of the color correction pipeline1,3 aims to render the acquired image as closely as possible to what a human observer would have perceived if placed in the original scene, emulating the color constancy feature of the human visual system (HVS), i.e., the ability of perceiving relatively constant colors when objects are lit by different illuminants.4 The illuminant estimation is an ill-posed problem,5 and it is one of the most delicate modules of the entire image processing pipeline. Numerous methods exist in the literature, and excellent reviews of them can be found in the works of Hordley,4 Bianco et al.,6 and Gijsenij et al.7 A recent research area, which has shown promising results, aims to improve illuminant estimation by using visual information automatically extracted from the images. The existing algorithms exploit both low-level,8,9 intermediate-level,10 and high-level11,12 visual information.

The second stage of the color correction pipeline transforms the image data into a standard color space. This transformation, usually called color matrixing, is needed because the spectral sensitivity functions of the sensor color channels rarely match those of the desired output color space. This transformation is usually performed by using a linear transformation matrix, and it is optimized assuming that the illuminant in the scene has been successfully estimated and compensated for Refs. 13 and 14.

Both the illuminant estimation process and the color correction matrix concur in the formation of the overall perceived image quality. These two processes have always been studied and optimized separately, thus ignoring the interactions between them with the only exception (to the best of our knowledge) of the authors that have investigated their interactions and how to optimize them for the overall color accuracy on datasets synthetically generated.6 Nevertheless many factors on real systems can affect the color fidelity, e.g., noise, limited dynamic range, and signal degradation due to digital signal processing. This work aims to design and test new and more robust color correction pipelines. The first module exploits different illuminant estimation and correction algorithms that are automatically selected on the basis of the image content. The second module, taking into account the behavior of the first module, makes it possible to alleviate error propagation and improve color rendition accuracy. The proposed pipelines are tested and compared with state of the art solutions on a publicly available dataset of RAW images.15 Experimental results show that illuminant estimation algorithms exploiting visual information extracted from the images, and taking into account the cross-talks between the modules of the pipeline can significantly improve color rendition accuracy.

An image acquired by a digital camera can be represented as a function ρ mainly dependent on three physical factors: the illuminant spectral power distribution I(λ), the surface spectral reflectance S(λ), and the sensor spectral sensitivities C(λ). Using this notation, the sensor responses at the pixel with coordinates (x,y) can be thus described as Display Formula

ρ(x,y)=ωI(λ)S(x,y,λ)C(λ)dλ,(1)
where ω is the wavelength range of the visible light spectrum, ρ and C(λ) are three-component vectors. Since the three sensor spectral sensitivities are usually more sensitive, respectively to the low, medium, and high wavelengths, the three-component vector of sensor responses ρ=(ρ1,ρ2,ρ3) is also referred to as the sensor or camera raw RGB=(R,G,B) triplet. In the following, we adopt the convention that RGB triplets are represented by column vectors.

In order to render the acquired image as close as possible to what a human observer would perceive if placed in the original scene, the first stage of the color correction pipeline aims to emulate the color constancy feature of the HVS, i.e., the ability to perceive relatively constant colors when objects are lit by different illuminants. The dedicated module is usually referred to as automatic white balance (AWB), which should be able to determine from the image content the color temperature of the ambient light and compensate for its effects. Numerous methods exist in the literature, and Hordley,4 Bianco et al.,6 and Gijsenij et al.7 give an excellent review of them. Once the color temperature of the ambient light has been estimated, the compensation for its effects is generally based on the Von Kries hypothesis,16 which states that color constancy is an independent regulation of the three cone signals through three different gain coefficients. This correction can be easily implemented on digital devices as a diagonal matrix multiplication.

The second stage of the color correction pipeline transforms the image data into a standard RGB (e.g., sRGB, ITU-R BT.709) color space. This transformation, usually called color matrixing, is needed because the spectral sensitivity functions of the sensor color channels rarely match those of the desired output color space. Typically this transformation is a 3×3 matrix with nine variables to be optimally determined, and there are both algebraic13 and optimization-based methods14,16 to find it.

The typical color correction pipeline can be thus described as follows: Display Formula

[RGB]out=(α[a11a12a13a21a22a23a31a32a33][rawbgawbbawb][RGB]in)γ,(2)
where RGBin are the camera raw RGB values, α is an exposure compensation common gain, the diagonal matrix diag(rawb,gawb,bawb) is the channel-independent gain compensation of the illuminant, the full 3×3 matrix a(i,j), (i,j)={1,2,3}2 is the color space conversion transform from the device-dependent RGB to the sRGB color space, γ is the gamma correction defined for the sRGB color space, and RGBout are the output sRGB values. The exponent γ is intended here, with abuse of notation, to be applied to every element of the matrix.

The color correction pipeline is usually composed of a fixed white balance algorithm, coupled with a color matrix transform optimized for a single illuminant.

In this work, since it has previously been shown5 that within a set of AWB algorithms, the best and the worst ones do not exist, but they change on the basis of the image characteristic, we consider a set of single AWB algorithms,17 and two classification-based modules,10,8 able to identify the best AWB algorithm to use for each image exploiting automatically extracted information about the image class or image content in terms of low-level features.

For what concerns the matrix transform module, we consider together with a single matrix optimized for a single illuminant a module based on multiple matrices optimized for different illuminants, and we consider matrices optimized taking into account the AWB algorithm behavior.6

Automatic White Balance Modules

The first AWB module considered is the best single (BS) algorithm extracted from the ones proposed by Van de Weijer et al.17 They unified a variety of color constancy algorithms in a unique framework. The different algorithms estimate the color I of the illuminant by implementing instantiations of the following equation: Display Formula

I(n,p,σ)=1k(nρσ(x,y)|pdxdy)1/p,(3)
where n is the order of the derivative, p is the Minkowski norm, ρσ(x,y)=ρσ(x,y)Gσ(x,y) is the convolution of the image with a Gaussian filter Gσ(x,y) with scale parameter σ, and k is a constant to be chosen so that the illuminant color I has unit length. Varying the three variables (n,p,σ), it is possible to generate six algorithm instantiations that correspond with well-known and widely used color constancy algorithms. The six different instantiations used in this work are reported in Table 1, together with their underlying assumptions.

Table Grahic Jump Location
Table 1Color constancy algorithms that can be generated as instantiations of Eq. (3), together with their underlying hypothesis.

The different AWB instances have been optimized on the dataset proposed by Ciurea and Funt,18 and the best performing one on an independent training set is selected as the BS algorithm.10 The optimal parameters found are reported in Table 1.

The second AWB module considered is the class based (CB) algorithm extracted from 10. It adopts a classification step to assign each image to indoor, outdoor, or ambiguous classes. The classifier, which is described below, is trained on general purpose, low-level features automatically extracted from the images: color histogram, edge direction histogram, statistics on the wavelet coefficients, and color moments (see 10 for a more detailed description of them). A different AWB algorithm is associated to each of the three possible classes: on the basis of the classification result, only the corresponding AWB algorithm selected is applied. The strategy for the selection and the tuning of the most appropriate algorithm (or combination of algorithms) for each class is fully described.10 The block diagram of the CB algorithm is reported in Fig. 1.

Grahic Jump LocationF1 :

Block diagram of the class based (CB) algorithm.

The third module considered is the feature based (FB) algorithm extracted from 8. It is based on five independent AWB algorithms and a classification step that automatically selects which AWB algorithm to use for each image. It is also possible to use the output of the classifier as weights to combine the estimations of the different algorithms considered.5,19 The classifier is trained on low-level features automatically extracted from the images. The feature set includes the general purpose features used for the CB algorithm and some features specifically designed. These features are the number of different colors, the percentage of color components that are clipped to the highest and lowest value that can be represented in the image color space, the magnitudes of the edges, and a cast index representing the extent of the presence of a color cast in the image (inspired by the work done in 20). See 8 for a detailed description of the features. The block diagram of the FB algorithm is reported in Fig. 2.

Grahic Jump LocationF2 :

Block diagram of the feature based (FB) algorithm.

The CB and FB AWB modules share the same classification strategy. They use tree classifiers constructed according to the CART methodology.21 Briefly, tree classifiers are classifiers produced by recursively partitioning the predictors space, each split being formed by conditions related to the predictors values. In tree terminology, subsets are called nodes: the predictors space is the root node, terminal subsets are terminal nodes, and so on. Once a tree has been constructed, a class is assigned to each of the terminal nodes, and when a new case is processed by the tree, its predicted class is the class associated with the terminal node into which the case finally moves on the basis of its predictors values. The construction process is based on training sets of cases of known class. Tree classifiers compare well with other consolidated classifiers. Many simulation studies have shown their accuracy to be very good, often close to the achievable optimum.21 Moreover, they provide a clear understanding of the conditions that drive the classification process. Finally, they imply no distributional assumptions for the predictors and can handle both quantitative and qualitative predictors in a natural way. Since in high dimensional and complex problems, as is the case here, it is practically impossible to obtain in one step good results in terms of accuracy, no matter how powerful the chosen class of classifiers is, we decided to perform the classification by also using what is called a “perturbing and combining” method.22,23 Methods of this kind, which generate in various ways multiple versions of a base classifier and use these to derive an aggregate classifier, have proved successful in improving accuracy. We used bagging (bootstrap aggregating), since it is particularly effective when the classifiers are unstable, as trees are; that is, when small perturbations in the training sets, or in the construction process of the classifiers, may result in significant changes in the resulting prediction. With bagging the multiple versions of the base classifier are formed by making bootstrap replicates of the training set and using them as new training sets. The aggregation is made by majority vote. In any particular bootstrap replicate, each element of the training set may appear repeated, or not at all, since the replicates are obtained by resampling with replacement. To provide a measure of confidence in the classification results and, still, greater accuracy, we applied an ambiguity rejection rule24 to the bagged classifier: the classification obtained by means of the majority vote is rejected if the percentage of trees that contribute to it is lower than a given threshold. In this way only those results to which the classifier assigns a given confidence, as set by the threshold, are accepted.

Color Matrix Transform Modules

The color matrix transform modules considered are extracted from the strategies proposed by the authors in 6. In the following a more compact version of Eq. (2) is used: Display Formula

RGBout=(αAIW·RGBin)γ,(4)
where α, IW and A, respectively represent the exposure compensation gain, the diagonal matrix for the illuminant compensation and the color matrix transformation.

The first color matrix transform module considered is named Single ILLuminant (SILL) since it is based on a single matrix transform optimized for a single illuminant. Given a set of n different patches whose sRGB values r are known, and the corresponding camera raw values c measured by the sensor when the patches are lit by the chosen illuminant, what is usually done is to find the matrix M that satisfies Display Formula

M=arg(minAR3×3k=1nE(rk,(αAIWck)γ)),(5)
where E(·) is the chosen error metric, and the subscript k indicates the triplet in the k’th column of the matrix, i.e., it indicates the tri-chromatic values of the k’th patch. In this work, the error metric E adopted is the average ΔE94 colorimetric error between the reference and calculated sRGB values mapped in the CIEL*a*b* color space. The values of α and IW have been previously computed in order to perfectly expose the scene and compensate for the illuminant. Given the importance of neutral tones in the color reproduction, the 9 deg of freedom of the color matrix transformation are usually reduced to six by a white point preserving constraint, i.e., a neutral color in the device-dependent color space should be mapped to a neutral color in the device independent color space. This can be easily obtained by constraining each row to sum to one.

The second color matrix transform module considered is named multiple illuminant (MILL). It differs from the first module since it is based on multiple matrix transforms, with each one optimized for a different illuminant by using Eq. (5). Therefore for each image a different matrix transform is used. First of all the AWB algorithm is applied to estimate the illuminant compensation gains, then the two training illuminants ILLi and ILLj with the most similar gains are identified, and the matrix transform is calculated as follows: Display Formula

M=αMILLi+(1α)MILLj,(6)
where Display Formula
α=D(gains,gainsj)D(gains,gainsi)+D(gains,gainsj),(7)
and D is the angular error between the gains considered, i.e., Display Formula
D=arccos(gains1T·gains2gains1·gains2).(8)

The reference illuminants could be a set of predefined standard illuminants (as done in 6), or it could be found by clustering the ground truth illuminants of the images in the training set. Then, for each centroid of the clusters found, the best color correction matrix is computed. The latter approach is here adopted as described in Sec. 5.

The third color matrix transform module considered is named SILL with white balance error buffer (SILLWEB). It is based on a single matrix transform optimized for a single illuminant, taking into account the behavior of the AWB module used. Suppose the best gain coefficients g0=[r0,g0,b0] have already been determined and reshaped in the diagonal transform G0 to compensate the considered illuminant; we then generate a set g={g1,,gs} of s gain coefficients with different distances from g0, measured using the ΔE94 error metric. These can be used to simulate errors that may occur in the AWB process and are paired with a weights distribution u={u0,,us} that reflects the frequency of the considered errors. The optimization problem can be thus formulated as Display Formula

M=arg(minAR3×3j=0suj(k=1nE(rk,(αjAGjck)γ)))subjecttoj=13A(i,j)=1,i{1,2,3},(9)
where Gj, j={0,,s} are the diagonal matrices obtained, respectively, by reshaping the gain coefficients {g0,,gs}.

The fourth color matrix transform module considered is named MILL with white balance error buffer (MILLWEB). It differs from the third module since it is based on multiple matrix transforms, with each one optimized for a different taking illuminant using Eq. (9). For each image a different matrix transform is used. First the AWB algorithm is applied to estimate the illuminant compensation gains, then the two training illuminants ILLi and ILLj with the most similar gains are identified, and the matrix transform is calculated as in Eqs. (6) and (7).

All the matrices for the different color matrix transform modules are found by optimization using the pattern search method (PSM). PSMs are a class of direct search methods for nonlinear optimization.25 PSMs are simple to implement and do not require any explicit estimate of derivatives. Furthermore, global convergence can be established under certain regularity assumptions of the function to minimize.26

The general form of a PSM is reported in Table 2, where f is the function to be minimized, k is the iteration number, xk is the current best solution, Dk is the set of search directions, and Δk is a step-length parameter.

Table Grahic Jump Location
Table 2Pseudo-code of the general form of a pattern search method (PSM).

The aim of this section is to investigate how the proposed methods can be combined in order to design a new color correction pipeline. In particular, we investigate the color accuracy improvement that the illuminant estimation algorithms of Sec. 3 and the color space conversion strategies of Sec. 4 can give when they are used individually and combined properly.

Image Dataset and Evaluation Procedure

To test the performance of the investigated processing pipelines, a standard dataset of RAW camera images having a known color target is used.15 This dataset is captured using a high-quality digital SLR camera in RAW format and is therefore free of any color correction. This dataset was originally available in sRGB-format, but Shi and Funt27 reprocessed the raw data to obtain linear images with a higher dynamic range (12 bits as opposed to standard 8 bits). The dataset consists of a total of 568 images. The Macbeth ColorChecker (MCC) chart is included in every scene acquired, and this allows us to estimate accurately the actual illuminant of each acquired image.27 Some examples from the image dataset are shown in Fig. 3. The spatial coordinates of the MCC in each image of the dataset have been automatically detected28 and manually refined.

Grahic Jump LocationF3 :

Typical examples of the images belonging to the RAW dataset used.

The flowchart of the evaluation procedure adopted is given in Fig. 4, where it can be seen that the only step in which the MCC chart is cropped is the illuminant estimation one.

Grahic Jump LocationF4 :

Pipeline evaluation: the Macbeth ColorChecker (MCC) is localized and masked; the illuminant is estimated on the image with the masked MCC, and the unmasked image is illuminant-corrected on the basis of this estimate; the color correction matrix is then applied to this image, the RGB coordinates of the MCC are extracted and compared with the MCC theoretical ones.

The investigated illuminant estimation algorithms described in Sec. 3 have been individually applied to the images of the dataset, excluding the MCC chart regions that have been previously cropped. Given these estimations, the illuminant corrections are then performed on the whole images (therefore also including the MCC chart).

The color matrix transformations found according to the computational strategies described in Sec. 4 are then applied to the whole, white balanced images. For each processed image, the MCC chart is then extracted, and the average RGB values of the central area of each patch are calculated. The color rendition accuracy of the pipeline is measured in terms of average ΔE94 error between the CIEL*a*b* color coordinates of the color corrected MCC patches, and their theoretical CIEL*a*b* values that are computed using standard equations from their theoretical RGB values.

In this section the color correction pipelines composed of the combination of the modules for the illuminant estimation algorithms of Sec. 3 and the color matrix transform modules of Sec. 4 are tested. Globally, 20 different pipelines have been tested; they are generated as an exhaustive combination of the modules proposed. The acronyms of the proposed strategies are generated using the scheme reported in Fig. 5.

Grahic Jump LocationF5 :

Composition rules for the generation of the acronyms of the strategies compared.

The first part indicates the typology of AWB used (BS=BestSingle: the BS AWB algorithm among the general purpose ones considered in 10 is used; CB=Class-Based: the algorithm described in 10, based on an indoor-outdoor classification is used; FB=Feature-Based: the algorithm described in 8 is used, which is based on five independent AWB algorithms and a classification step that automatically selects which AWB algorithm to use for each image). The second part indicates the number and type of the color correction matrix used (SILL=singleilluminant: single matrix optimized for a fixed single illuminant, MILL=multipleilluminant: multiple matrices each optimized for a different single illuminant). The third part indicates if the strategy implements color correction matrices able to compensate for AWB errors (WEB=WhitebalanceErrorBuffer) or not (0). The last part indicates if a real classifier has been used in the AWB module (0) or manual classification (ideal). The symbol 0 is reported in the scheme but is intended as the null character and thus omitted in the acronyms generated.

Since the considered modules need a training phase, 30% of the images in the dataset were randomly selected and used as training set; the remaining 70% were used as test set. For the strategies that are based on multiple color correction matrices, these are computed by first clustering the ground truth illuminants of the images in the training set into seven different clusters using a k-means algorithm.29 Then, for each centroid of the clusters found, the best color correction matrix is calculated.

The average over the test images of the average ΔE94 colorimetric errors obtained by the tested pipelines on the MCCs are computed and reported in Table 3. The average of the maximum ΔE94 colorimetric error are also reported. For both the error statistics the percentage improvement over the baseline method, i.e., BS-SILL pipeline, are reported. The results of the pipelines tested are clustered into four different groups depending on the color correction strategy adopted.

Table Grahic Jump Location
Table 3Color correction pipeline accuracy comparison.

To understand if the differences in performance among the pipelines considered are statistically significant, we have used the Wilcoxon signed-rank test.30 This statistical test permits comparison of the whole error distributions without limiting to punctual statistics. Furthermore, it is well suited because it does not make any assumptions about the underlying error distributions, and it is easy to find, using for example the Lilliefors test,31 that the assumption about the normality of the error distributions does not always hold. Let X and Y be random variables representing the ΔE94 errors obtained on the MCCs of all test images by two different pipelines. Let μX and μY be the median values of such random variables. The Wilcoxon signed-rank test can be used to test the null hypothesis H0:μX=μY against the alternative hypothesis H1:μXμY. We can test H0 against H1 at a given significance level α. We reject H0 and accept H1 if the probability of observing the error differences we obtained is less than or equal to α. We have used the alternative hypothesis H1:μX<μY with a significance level α=0.05. Comparing the error distributions of each pipeline with all the others gives the results reported in Table 4. A “+” sign in the (i,j) position of the table means that the error distribution obtained with the pipeline i has been considered statistically better than that obtained with the pipeline j; a “−” sign that it has been considered statistically worse, and a “=“ sign that they have been considered statistically equivalent.

Table Grahic Jump Location
Table 4Wilcoxon signed rank test results on the error distributions obtained by the different pipelines. A “+” sign in the (i, j)-position means that the error distribution obtained with the pipeline i has been considered statistically better than that obtained with the pipeline j, a “−” sign that it has been considered statistically worse, and an “=” sign that they have been considered statistically equivalent.

It is possible to note in Table 3 that in all the groups of pipelines proposed (groups that share the same color correction matrix strategy, i.e., SILL, MILL, SILLWEB, and MILLWEB), the use of the FB AWB leads to a higher color-rendition accuracy with respect to the use of the CB AWB and of the BS AWB. It is interesting to notice that significant improvements in the color rendition accuracy can be achieved even if the classifiers used for the feature and CB AWB strategies are not optimal.

Analyzing the behavior of the pipelines sharing the same AWB strategy, it is possible to notice that the results of 6 are also confirmed. In fact, the multiple illuminant color correction (MILL) performs better than the single illuminant one (SILL). The single illuminant color correction, which is optimized taking into account the statistics of how the AWB algorithm tends to make errors (SILLWEB), performs better than the multiple illuminant color correction (MILL). Finally, the multiple illuminant color correction with white balance error buffer (MILLWEB) performs better than the corresponding single illuminant instantiation (SILLWEB).

For what concerns the best pipeline proposed, which is the FB-MILLWEB, it can be observed that when using the ideal classifier, 48% of improvement (from 0% to 14.15% with respect to the benchmarking pipeline) is due to the use of the MILLWEB color correction matrix approach and the remaining 52% (from 14.15% to 29.32% with respect to the benchmarking pipeline) to the FB AWB approach. When using the real classifier the remaining part to the FB AWB approach. This can be considered a lower bound of the pipeline performance, since, as already explained before, the classifier used is not optimal for the image database used.

In Fig. 6 the workflow of the best performing pipeline proposed, i.e., the FB-MILLWEB, is reported. The low-level features considered are extracted from the RAW image and fed to the classifier, which as output gives the weights to use for the linear combination of the five simple AWB algorithms considered. The AWB correction gains given by the five simple AWB algorithms considered are then combined to give the AWB correction gains to use. The image is then corrected with these gains to obtain an AWB corrected image. The correction gains are used to identify the two training illuminants most similar to the estimated one. The two color correction matrices computed for the two identified illuminants are retrieved and combined accordingly to the illuminant similarity. The image is finally color corrected using this color correction matrix.

Grahic Jump LocationF6 :

Workflow of the pipeline composed of the feature-based (FB) automatic white balance (AWB) and the multiple illuminant color matrixing with white balance error buffer (FB-MILLWEB).

In Fig. 7 it is shown the image on which the best pipeline proposed, i.e., FB-MILLWEB, makes the larger color error. For sake of comparison, the results obtained with BS-SILL (a); FB-SILL (b); FB-SILLWEB (c); and FB-MILLWEB (d) pipelines are also shown. Finally in Fig. 7(e) the best achievable results is reported, which is computed using the achromatic patches of the MCC to estimate the ideal AWB gains, and the color space transformation has been optimized specifically for this image. Taking into account that the image reproduction may have altered the image content, making the differences between the different pipelines not clearly appreciable, we report in Fig. 8 the error distribution between the ideal color-corrected image [Fig. 7(e)] and the output of the tested pipelines. The images reported in Fig. 7(a)7(d) are therefore converted in CIEL*a*b* color coordinates and the ΔE94 colorimetric error is computed for every pixel of each image with respect to the ideal image. The histograms of the colorimetric errors are, respectively reported in Fig. 8(a)8(d). To compare the color error distributions of the pipelines considered on this image, we use the Wilcoxon signed-rank test. The output of the test is reported in Table 5. It is possible to notice that even in the worst case example, the FB-MILLWEB pipeline, which is the best on the whole dataset, is still the best one.

Grahic Jump LocationF7 :

Worst case example. The results obtained by different pipelines are reported: BS-SILL (a); FB-SILL (b); FB-SILLWEB (c); FB-MILLWEB (d); and ideal correction (e).

Grahic Jump LocationF8 :

Histograms of the ΔE94 colorimetric error for the images reported in Figs. 7(a)–(d), using Figs. 7(e) as reference.

Table Grahic Jump Location
Table 5Wilcoxon signed rank test results on the error distributions obtained by the different pipelines on the worst case example.

Digital camera sensors are not perfect and do not encode colors the same way in which the human eye does. A processing pipeline is thus needed to convert the RAW image acquired by the camera to a representation of the original scene that should be as faithful as possible. In this work we have designed and tested new color correction pipelines, which exploit the cross-talks between its modules in order to lead to a higher color rendition accuracy. The effectiveness of the proposed pipelines is shown on a publicly available dataset of RAW images. The experimental results show that in all the groups of pipelines proposed (groups that share the same color correction matrix strategy, i.e., SILL, MILL, SILLWEB, and MILLWEB), the use of the FB AWB leads to a higher color-rendition accuracy when compared with the use of the CB AWB and of the BS AWB. It is interesting to note that significant improvements in the color-rendition accuracy can be achieved even if the classifiers used for the feature and CB AWB strategies are not optimal. Analyzing the behavior of the pipelines sharing the same AWB strategy, it is possible to note that the results of 6 are also confirmed. In fact the multiple illuminant color correction (MILL) performs better than the single illuminant one (SILL). The single illuminant color correction, which is optimized taking into account the statistics of how the AWB algorithm tends to make errors (SILLWEB), performs better than the multiple illuminant color correction (MILL). Finally, the multiple illuminant color correction with white balance error buffer (MILLWEB) performs better than the corresponding single illuminant instantiation (SILLWEB).

The present work makes it also possible to identify some open issues that must be addressed in the future. Illuminant estimation algorithms are generally based on the simplifying assumption that the spectral distribution of a light source is uniform across scenes. However, in reality, this assumption is often violated due to the presence of multiple light sources.32 Some multiple illuminant estimation algorithms have been proposed;33 however, they assume very simple setups, or a knowledge of the number and color of the illuminants.34

Once the scene illuminant has been estimated, the scene is usually corrected in the RGB device dependent color space using the diagonal Von Kries model.16 Several studies have investigated the use of different color spaces for the illuminant correction35,36 as well as nondiagonal models.37 A different approach could be to use chromatic adaptation transforms38,39 to correct the scene illuminant.

As suggested by a reviewer, the use of larger color correction matrices should be investigated, taking into account not only color rendition accuracy, but also noise amplification in particular on dark colors.

Ramanath  R. et al., “Color image processing pipeline,” IEEE Signal Proc. Mag.. 22, (1 ), 34 –43 (2005). 1053-5888 CrossRef
Weerasinghe  C. et al., “Novel color processing architecture for digital cameras with CMOS image sensors,” IEEE Trans. Consum. Electron.. 51, (4 ), 1092 –1098 (2005). 0098-3063 CrossRef
Kao  W.-C. et al., “Design consideration of color image processing pipeline for digital cameras,” IEEE Trans. Consum. Electron.. 52, (4 ), 1144 –1152 (2006). 0098-3063 CrossRef
Hordley  S. D., “Scene illuminant estimation: past, present, and future,” Color Res. Appl.. 31, (4 ), 303 –314 (2006). 0361-2317 CrossRef
Bianco  S., Gasparini  F., Schettini  R., “Consensus based framework for illuminant chromaticity estimation,” J. Electron. Imag.. 17, (2 ), 023013  (2008). 1017-9909 CrossRef
Bianco  S. et al., “Color space transformations for digital photography exploiting information about the illuminant estimation process,” J. Opt. Soc. Am. A. 29, (3 ), 374 –384 (2012). 0740-3232 CrossRef
Gijsenij  A., Gevers  T., van de Weijer  J., “Computational color constancy: survey and experiments,” IEEE Trans. Image Proc.. 20, (9 ), 2475 –2489 (2011). 1057-7149 CrossRef
Bianco  S. et al., “Automatic color constancy algorithm selection and combination,” Pattern Recogn.. 43, (3 ), 695 –705 (2010). 0031-3203 CrossRef
Gijsenij  A., Gevers  T., “Color constancy using natural image statistics and scene semantics,” IEEE Trans. Pattern Anal. Mach. Intell.. 33, (4 ), 687 –698 (2011). 0162-8828 CrossRef
Bianco  S. et al., “Improving color constancy using indoor-outdoor image classification,” IEEE Trans. Image Proc.. 17, (12 ), 2381 –2392 (2008). 1057-7149 CrossRef
van de Weijer  J., Schmid  C., Verbeek  J., “Using high-level visual information for color constancy,” in  Proc. IEEE 14th Int. Conf. on Computer Vision , pp. 1 –8,  IEEE ,   Rio de Janeiro  (2007).
Bianco  S., Schettini  R., “Color constancy using faces,” in  Proc. IEEE Conf. Computer Vision and Pattern Recognit. , pp. 65 –72,  IEEE ,  Providence, Rhode Island  (2012).
Hubel  P. M. et al., “Matrix calculations for digital photography,” in  Proc. 5th Color Imaging Conf.: Color Science, Systems and Applications , pp. 105 –111,  GrUVi ,  Scottsdale, Arizona  (1997).
Bianco  S. et al., “A new method for RGB to XYZ transformation based on pattern search optimization,” IEEE Trans. Consum. Electron.. 53, (3 ), 1020 –1028 (2007). 0098-3063 CrossRef
Gehler  P. V. et al., “Bayesian color constancy revisited,” in  Proc. IEEE Computer Society Conf. on Computer Vision and Pattern Recognit. , pp. 1 –8,  IEEE ,  Anchorage, Alaska  (2008).
von Kries  J., Chromatic Adaptation. ,  Festschrift der Albrecht-Ludwig-Universitat ,  Fribourg  (1902); [Translation: MacAdam  D. L., Sources of Color Science. ,  MIT Press ,  Cambridge  (1970)].
van de Weijer  J., Gevers  T., Gijsenij  A., “Edge-based color constancy,” IEEE Trans. Image Proc.. 16, (9 ), 2207 –2214 (2007). 1057-7149 CrossRef
Ciurea  F., Funt  B., “A large image database for color constancy research,” in  Proc. IS&T/SID 11th Color Imaging Conf. , pp. 160 –164 (2003).
Bianco  S., Gasparini  F., Schettini  R., “Combining strategies for white balance,” Proc. SPIE. 6502, , 65020D  (2007). 0277-786X CrossRef
Gasparini  F., Schettini  R., “Color balancing of digital photos using simple image statistics,” Pattern Recogn.. 37, (6 ), 1201 –1217 (2004). 0031-3203 CrossRef
Breiman  L. et al., Classification and Regression Trees. ,  Wadsworth and Brooks/Cole ,  Belmont, California  (1984).
Breiman  L., “Bagging predictors,” Mach. Learn.. 26, (2 ), 123 –140 (1996). 0885-6125 
Breiman  L., “Arcing classifiers,” Ann. Stat.. 26, (3 ), 801 –849 (1998). 0090-5364 CrossRef
Vailaya  A., Jain  A., “Reject option for VQ-based Bayesian classification,” in  Proc. 15th Int. Conf. Pattern Recognit. , Vol. 2, pp. 48 –51,  IEEE ,  Barcelona, Spain  (2000).
Lewis  R. M., Torczon  V., “Pattern search methods for linearly constrained minimization,” SIAM J. Optim.. 10, (3 ), 917 –941 (2000). 1095-7189 CrossRef
Lewis  R. M., Torczon  V., “On the convergence of pattern search algorithms,” SIAM J. Optim. 7, (1 ), 1 –25 (1997). 1095-7189 CrossRef
Shi  L., Funt  B. V., Re-processed version of the Gehler color constancy database of 568 images, http://www.cs.sfu.ca/colour/data/ (5  July 2012).
Bianco  S., Cusano  C., “Color target localization under varying illumination conditions,” in  Proc. 3rd Int. Workshop on Computational Color Imaging , Lecture Notes in Computer Science , Vol. 6626, pp. 245 –255,  Springer ,  Berlin, Heidelberg  (2011).
Hartigan  J. A., Wong  M. A., “Algorithm AS136: A k-means clustering algorithm,” Appl. Stat.. 28, , 100 –108 (1979).CrossRef
Wilcoxon  F., “Individual comparisons by ranking methods,” Biometrics. 1, (6 ), 80 –83 (1945). 0006-341X CrossRef
Lilliefors  H., “On the Kolmogorov-Smirnov test for normality with mean and variance unknown,” J. Am. Stat. Assoc.. 62, (318 ), 399 –402 (1967). 0003-1291 CrossRef
Gijsenij  A., Lu  R. , Gevers  T., “Color constancy for multiple light sources,” IEEE Trans. Image Proc.. 21, (2 ), 697 –707 (2012). 1057-7149 CrossRef
Ebner  M., Color Constancy. ,  John Wiley & Sons ,  England  (2007).
Hsu  E. et al., “Light mixture estimation for spatially varying white balance,” in  Proc. ACM SIGGRAPH ,  ACM ,  New York  (2008).
Barnard  K., Ciurea  F., Funt  B., “Sensor sharpening for computational color constancy,” J. Opt. Soc. Am. A. 18, (11 ), 2728 –2743 (2001). 0740-3232 CrossRef
Xiao  F. et al., “Preferred color spaces for white balancing,” Proc. SPIE. 5017, , 342 –350 (2003). 0277-786X CrossRef
Funt  B., Jiang  H., “Nondiagonal color correction,” in  Proc. Int. Conf. Image Process. , Vol. 1, pp. 481 –484,  IEEE  (2003).
CIE 160:2004, “A review of chromatic adaptation transforms,” ISBN: 9783901906305 (2004).
Bianco  S., Schettini  R., “Two new von Kries based chromatic adaptation transforms found by numerical optimization,” Color Res. App.. 35, (3 ), 184 –192 (2010). 0361-2317 CrossRef

Grahic Jump LocationImage not available.

Simone Bianco obtained a PhD in computer science at DISCo (Dipartimento di Informatica, Sistemistica e Comunicazione) of the University of Milano-Bicocca, Italy, in 2010. He obtained BSc and the MSc degrees in mathematics from the University of Milano-Bicocca, Italy, respectively, in 2003 and 2006. He is currently a postdoc working on image processing. The main topics of his current research concern digital still cameras processing pipelines, color space conversions, optimization techniques, and characterization of imaging devices.

Grahic Jump LocationImage not available.

Arcangelo R. Bruna received a master’s degree from Palermo University, Italy, in 1998. He works for ST Microelectronics, and his research interests are in the area of image processing (noise reduction, auto white balance, video stabilization, etc.) and computer vision algorithms (visual search). He is also currently involved in the MPEG-CDVS (compact descriptors for visual search) standardization.

Grahic Jump LocationImage not available.

Filippo Naccari received his MS Italian degree in electronic engineering at University of Palermo, Italy. Since July 2002, he has been working at STMicroelectronics as researcher at the Advanced System Technology, Computer Vision Group, Catania Lab, Italy. His research interests are in the field of digital color and image processing, color constancy, and digital images coding.

Grahic Jump LocationImage not available.

Raimondo Schettini is a professor at the University of Milano Bicocca (Italy). He is vice director of the Department of Informatics, Systems and Communication, and head of Imaging and Vision Lab (www.ivl.disco.unimib.it). He has been associated with Italian National Research Council (CNR) since 1987 where he has headed the Color Imaging lab from 1990 to 2002. He has been team leader in several research projects and published more than 250 reference papers and six patents about color reproduction and image processing, analysis, and classification. He has been recently elected fellow of the International Association of Pattern Recognition (IAPR) for his contributions to pattern recognition research and color image analysis.

© The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Citation

Simone Bianco ; Arcangelo R. Bruna ; Filippo Naccari and Raimondo Schettini
"Color correction pipeline optimization for digital cameras", J. Electron. Imaging. 22(2), 023014 (May 14, 2013). ; http://dx.doi.org/10.1117/1.JEI.22.2.023014


Figures

Grahic Jump LocationF1 :

Block diagram of the class based (CB) algorithm.

Grahic Jump LocationF2 :

Block diagram of the feature based (FB) algorithm.

Grahic Jump LocationF6 :

Workflow of the pipeline composed of the feature-based (FB) automatic white balance (AWB) and the multiple illuminant color matrixing with white balance error buffer (FB-MILLWEB).

Grahic Jump LocationF7 :

Worst case example. The results obtained by different pipelines are reported: BS-SILL (a); FB-SILL (b); FB-SILLWEB (c); FB-MILLWEB (d); and ideal correction (e).

Grahic Jump LocationF8 :

Histograms of the ΔE94 colorimetric error for the images reported in Figs. 7(a)–(d), using Figs. 7(e) as reference.

Grahic Jump LocationF5 :

Composition rules for the generation of the acronyms of the strategies compared.

Grahic Jump LocationF4 :

Pipeline evaluation: the Macbeth ColorChecker (MCC) is localized and masked; the illuminant is estimated on the image with the masked MCC, and the unmasked image is illuminant-corrected on the basis of this estimate; the color correction matrix is then applied to this image, the RGB coordinates of the MCC are extracted and compared with the MCC theoretical ones.

Grahic Jump LocationF3 :

Typical examples of the images belonging to the RAW dataset used.

Tables

Table Grahic Jump Location
Table 1Color constancy algorithms that can be generated as instantiations of Eq. (3), together with their underlying hypothesis.
Table Grahic Jump Location
Table 3Color correction pipeline accuracy comparison.
Table Grahic Jump Location
Table 4Wilcoxon signed rank test results on the error distributions obtained by the different pipelines. A “+” sign in the (i, j)-position means that the error distribution obtained with the pipeline i has been considered statistically better than that obtained with the pipeline j, a “−” sign that it has been considered statistically worse, and an “=” sign that they have been considered statistically equivalent.
Table Grahic Jump Location
Table 5Wilcoxon signed rank test results on the error distributions obtained by the different pipelines on the worst case example.
Table Grahic Jump Location
Table 2Pseudo-code of the general form of a pattern search method (PSM).

References

Ramanath  R. et al., “Color image processing pipeline,” IEEE Signal Proc. Mag.. 22, (1 ), 34 –43 (2005). 1053-5888 CrossRef
Weerasinghe  C. et al., “Novel color processing architecture for digital cameras with CMOS image sensors,” IEEE Trans. Consum. Electron.. 51, (4 ), 1092 –1098 (2005). 0098-3063 CrossRef
Kao  W.-C. et al., “Design consideration of color image processing pipeline for digital cameras,” IEEE Trans. Consum. Electron.. 52, (4 ), 1144 –1152 (2006). 0098-3063 CrossRef
Hordley  S. D., “Scene illuminant estimation: past, present, and future,” Color Res. Appl.. 31, (4 ), 303 –314 (2006). 0361-2317 CrossRef
Bianco  S., Gasparini  F., Schettini  R., “Consensus based framework for illuminant chromaticity estimation,” J. Electron. Imag.. 17, (2 ), 023013  (2008). 1017-9909 CrossRef
Bianco  S. et al., “Color space transformations for digital photography exploiting information about the illuminant estimation process,” J. Opt. Soc. Am. A. 29, (3 ), 374 –384 (2012). 0740-3232 CrossRef
Gijsenij  A., Gevers  T., van de Weijer  J., “Computational color constancy: survey and experiments,” IEEE Trans. Image Proc.. 20, (9 ), 2475 –2489 (2011). 1057-7149 CrossRef
Bianco  S. et al., “Automatic color constancy algorithm selection and combination,” Pattern Recogn.. 43, (3 ), 695 –705 (2010). 0031-3203 CrossRef
Gijsenij  A., Gevers  T., “Color constancy using natural image statistics and scene semantics,” IEEE Trans. Pattern Anal. Mach. Intell.. 33, (4 ), 687 –698 (2011). 0162-8828 CrossRef
Bianco  S. et al., “Improving color constancy using indoor-outdoor image classification,” IEEE Trans. Image Proc.. 17, (12 ), 2381 –2392 (2008). 1057-7149 CrossRef
van de Weijer  J., Schmid  C., Verbeek  J., “Using high-level visual information for color constancy,” in  Proc. IEEE 14th Int. Conf. on Computer Vision , pp. 1 –8,  IEEE ,   Rio de Janeiro  (2007).
Bianco  S., Schettini  R., “Color constancy using faces,” in  Proc. IEEE Conf. Computer Vision and Pattern Recognit. , pp. 65 –72,  IEEE ,  Providence, Rhode Island  (2012).
Hubel  P. M. et al., “Matrix calculations for digital photography,” in  Proc. 5th Color Imaging Conf.: Color Science, Systems and Applications , pp. 105 –111,  GrUVi ,  Scottsdale, Arizona  (1997).
Bianco  S. et al., “A new method for RGB to XYZ transformation based on pattern search optimization,” IEEE Trans. Consum. Electron.. 53, (3 ), 1020 –1028 (2007). 0098-3063 CrossRef
Gehler  P. V. et al., “Bayesian color constancy revisited,” in  Proc. IEEE Computer Society Conf. on Computer Vision and Pattern Recognit. , pp. 1 –8,  IEEE ,  Anchorage, Alaska  (2008).
von Kries  J., Chromatic Adaptation. ,  Festschrift der Albrecht-Ludwig-Universitat ,  Fribourg  (1902); [Translation: MacAdam  D. L., Sources of Color Science. ,  MIT Press ,  Cambridge  (1970)].
van de Weijer  J., Gevers  T., Gijsenij  A., “Edge-based color constancy,” IEEE Trans. Image Proc.. 16, (9 ), 2207 –2214 (2007). 1057-7149 CrossRef
Ciurea  F., Funt  B., “A large image database for color constancy research,” in  Proc. IS&T/SID 11th Color Imaging Conf. , pp. 160 –164 (2003).
Bianco  S., Gasparini  F., Schettini  R., “Combining strategies for white balance,” Proc. SPIE. 6502, , 65020D  (2007). 0277-786X CrossRef
Gasparini  F., Schettini  R., “Color balancing of digital photos using simple image statistics,” Pattern Recogn.. 37, (6 ), 1201 –1217 (2004). 0031-3203 CrossRef
Breiman  L. et al., Classification and Regression Trees. ,  Wadsworth and Brooks/Cole ,  Belmont, California  (1984).
Breiman  L., “Bagging predictors,” Mach. Learn.. 26, (2 ), 123 –140 (1996). 0885-6125 
Breiman  L., “Arcing classifiers,” Ann. Stat.. 26, (3 ), 801 –849 (1998). 0090-5364 CrossRef
Vailaya  A., Jain  A., “Reject option for VQ-based Bayesian classification,” in  Proc. 15th Int. Conf. Pattern Recognit. , Vol. 2, pp. 48 –51,  IEEE ,  Barcelona, Spain  (2000).
Lewis  R. M., Torczon  V., “Pattern search methods for linearly constrained minimization,” SIAM J. Optim.. 10, (3 ), 917 –941 (2000). 1095-7189 CrossRef
Lewis  R. M., Torczon  V., “On the convergence of pattern search algorithms,” SIAM J. Optim. 7, (1 ), 1 –25 (1997). 1095-7189 CrossRef
Shi  L., Funt  B. V., Re-processed version of the Gehler color constancy database of 568 images, http://www.cs.sfu.ca/colour/data/ (5  July 2012).
Bianco  S., Cusano  C., “Color target localization under varying illumination conditions,” in  Proc. 3rd Int. Workshop on Computational Color Imaging , Lecture Notes in Computer Science , Vol. 6626, pp. 245 –255,  Springer ,  Berlin, Heidelberg  (2011).
Hartigan  J. A., Wong  M. A., “Algorithm AS136: A k-means clustering algorithm,” Appl. Stat.. 28, , 100 –108 (1979).CrossRef
Wilcoxon  F., “Individual comparisons by ranking methods,” Biometrics. 1, (6 ), 80 –83 (1945). 0006-341X CrossRef
Lilliefors  H., “On the Kolmogorov-Smirnov test for normality with mean and variance unknown,” J. Am. Stat. Assoc.. 62, (318 ), 399 –402 (1967). 0003-1291 CrossRef
Gijsenij  A., Lu  R. , Gevers  T., “Color constancy for multiple light sources,” IEEE Trans. Image Proc.. 21, (2 ), 697 –707 (2012). 1057-7149 CrossRef
Ebner  M., Color Constancy. ,  John Wiley & Sons ,  England  (2007).
Hsu  E. et al., “Light mixture estimation for spatially varying white balance,” in  Proc. ACM SIGGRAPH ,  ACM ,  New York  (2008).
Barnard  K., Ciurea  F., Funt  B., “Sensor sharpening for computational color constancy,” J. Opt. Soc. Am. A. 18, (11 ), 2728 –2743 (2001). 0740-3232 CrossRef
Xiao  F. et al., “Preferred color spaces for white balancing,” Proc. SPIE. 5017, , 342 –350 (2003). 0277-786X CrossRef
Funt  B., Jiang  H., “Nondiagonal color correction,” in  Proc. Int. Conf. Image Process. , Vol. 1, pp. 481 –484,  IEEE  (2003).
CIE 160:2004, “A review of chromatic adaptation transforms,” ISBN: 9783901906305 (2004).
Bianco  S., Schettini  R., “Two new von Kries based chromatic adaptation transforms found by numerical optimization,” Color Res. App.. 35, (3 ), 184 –192 (2010). 0361-2317 CrossRef

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Related Book Chapters

Topic Collections

Advertisement
  • Don't have an account?
  • Subscribe to the SPIE Digital Library
  • Create a FREE account to sign up for Digital Library content alerts and gain access to institutional subscriptions remotely.
Access This Article
Sign in or Create a personal account to Buy this article ($20 for members, $25 for non-members).
Access This Proceeding
Sign in or Create a personal account to Buy this article ($15 for members, $18 for non-members).
Access This Chapter

Access to SPIE eBooks is limited to subscribing institutions and is not available as part of a personal subscription. Print or electronic versions of individual SPIE books may be purchased via SPIE.org.