JEI Letters

Estimating reflectance from multispectral camera responses based on partial least-squares regression

[+] Author Affiliations
Hui-Liang Shen, Hui-Jiang Wan, Zhe-Chao Zhang

Zhejiang University, Department of Information and Electronic Engineering, Hangzhou 310027, China

J. Electron. Imaging. 19(2), 020501 (April 12, 2010). doi:10.1117/1.3385782
History: Received October 15, 2009; Revised March 14, 2010; Accepted March 17, 2010; Published April 12, 2010; Online April 12, 2010
Text Size: A A A

Open Access Open Access

In multispectral imaging systems, the accuracy of reflectance estimation can be degraded by the nonlinearity in imaging process, which is due to non-Gaussian distribution of the data and nonlinear optoelectronic conversion function of the camera. To deal with nonlinearity, we propose to extend camera responses by high-order polynomials and reduce the overfitting problem by partial least-squares (PLS) regression. Experiment shows that, in terms of both spectral and colorimetric error metrics, the proposed method performs better than Wiener estimation and ordinary polynomial regression, and is similar to polynomial regression with regularization.

Recently, high-fidelity color image reproduction has become increasingly important because of its many potential applications in textiles, medicine, digital archives, etc. To view objects under various illumination conditions, multispectral imaging has been extensively studied to estimate spectral reflectance of object surfaces.17 Multispectral images are usually acquired by trichromatic or monochrome cameras, accompanied by a set of color filters. The imaging process is often modeled by a linear system when the number of imaging channels is large. In recovery of spectral reflectance, Wiener estimation is deduced under the condition that the data conform to normal distribution.8 It works fairly well when the linearity condition is satisfied, and becomes a standard technique. However, it is likely that measured data are not in accordance to the normal distribution,23 and the optoelectronic conversion function (OECF) of the camera is nonlinear.9 Nonlinearity usually degrades the estimation accuracy of linear methods. Adaptive methods, which use local statistics instead of global ones, provide feasible solutions for improving accuracy.2,4 However, as adaptive methods usually need to recalculate the transform between responses and reflectance,4 they are computationally expensive and hence unsuitable for time-critical applications. An alternative way is to introduce nonlinear variables such as high-order polynomials. Hong, Luo, and Rhodes10 applied ordinary polynomial regression to predict colorimetric stimulus values from three-channel camera responses. The same technique has also been adopted in multispectral imaging.5 However, the extension of polynomial responses causes overfitting and collinearity problems when the number of imaging channels is large. Heikkinen et al. 6 introduced regularized polynomial modeling methods and a more general regularization framework for robust reflectance estimation.

We propose a global method for spectral estimation based on polynomial extension of camera responses and partial least-squares (PLS) regression.1112 The PLS is implemented in an iterative manner; its dimension (or number of PLS components) is determined by the spectral error distribution. The accuracy of the PLS method is compared with Wiener estimation and also polynomial regressions solved by ordinary least squares (OLS) and regularized least squares (RLS).

Suppose that the continuous visible spectrum is uniformly sampled at N (usually N=31) discrete wavelengths, and the number of imaging channels is C=16. Let F() be the OECF of camera, rRN×1 be the reflectance of imaged object surface, and MRC×N be the spectral responsivity of the imaging system, then camera response uRC×1 is computed asDisplay Formula

1u=F(Mr+n),
where nRC×1 denotes noise. When the camera behaves linearly, F(x)=x; otherwise it can be represented by high-order polynomials.9

By ignoring F(), the linear method tries to find a transform matrix WRN×C such that the estimate of reflectance can be computed asDisplay Formula

2r̂=Wu.
W can be calculated by Wiener estimation as4,7Display Formula
3WWE=KrMT(MKrMT+Kn)1,
where superscript T denotes the matrix transpose, and KrRN×N and KnRC×C are the autocorrelation matrices of reflectance r and noise n, respectively.

W can also be solved under the least-squares criterion. Let K be the number of training samples. We can construct reflectance matrix R=[r(1),r(2),,r(K)]RN×K and response matrix U=[u(1),u(2),,u(K)]RC×K, and then calculate the transform matrix as1,10Display Formula

4W=RU+,
where superscript + denotes the matrix pseudoinverse. The pseudoinverse operator in Eq. 4 and hereafter is numerically solved by singular value decomposition.13

Nonlinearity can degrade the performance of the linear reflectance estimation methods. It is natural to define a two-order polynomial response vector ũRJ×1 to deal with the nonlinearity due to OECF and non-Gaussian data distribution:Display Formula

5ũ=[1,u1,,uc,u12,u1u2,,u1uc,u22,u2u3,,u2uc,,uc1uc,uc2]T,
where uc(1cC) is the c’th element of u. As C=16, ũ has J=153 elements. By defining the polynomial response matrix as Ũ=[ũ(1),ũ(2),,ũ(K)]RJ×K, the transform matrix W can be solved under the OLS criterion asDisplay Formula
6WOLS=RŨ+,
and reflectance is predicted as r̂=WOLSũ.

Alternatively, W can also be computed by introducing a regularization (or penalization) term asDisplay Formula

7WRLS=(RŨT)(ŨŨT+ηI)1,
where η is a regularization parameter and I denotes the identity matrix. The overfitting problem can be diminished by the regularization term.6

As mentioned, the overfitting problem can occur in polynomial regression solved by OLS when the number of parameters in the mathematical model is greater than the number of dimensions of data variation. The overfitting problem is probably related to the increasing colinearity between the extended polynomial responses. In this regard, we propose to deal with it by dimensionality reduction using the PLS technique.11

The polynomial response matrix Ũ can be decomposed into a score matrix TRK×L and a loading matrix PRJ×L, with L being the number of PLS components, asDisplay Formula

8ŨT=TPT+E,
where ERK×J is a residual matrix. Similarity, R can also be decomposed into a score matrix DRK×L and a loading matrix QRN×L asDisplay Formula
9RT=DQT+F,
where FRK×N is a residual matrix.

The goal of PLS is to extract the common structure between ŨT and RT by searching a projection such that the covariance between the score matrices T and D is maximized. In matrix form, this relationship is written asDisplay Formula

10D=TB,
where BRL×L is the diagonal regression matrix.

The PLS algorithm is carried out in an iterative manner.12 To obtain an orthogonal score matrix T, a weight matrix GRJ×L is introduced in the iterative procedure. Let j be the iteration index, and g, t, q, and d be the j’th column vectors of matrices G, T, Q, and D, respectively. Before starting the iteration, let E=UT and F=RT. Matrices E and F are then column centered and normalized so that each variable has zero mean and unit variance. Let j=1 and d be any column of F, then compute Eqs. 1114 iteratively:Display Formula

11g=ETdETd,
Display Formula
12t=Eg,
Display Formula
13q=FTtFTt,
Display Formula
14d=Fq,
where ‖⋅‖ denotes the Euclidean norm. If t has not converged, return to Eq. 11, otherwise compute the value of b, which is the j’th diagonal element of matrix B, as b=dTttTt, and compute the factor loading, which is the j’th column vector of P, as p=ETttTt.

The residual matrices E and F needed for the next iteration are calculated asDisplay Formula

15E=EtpT,
andDisplay Formula
16F=FbtqT.

Note that Eqs. 1516 remove the variance associated with the obtained score and loading vectors before the next iteration. If j<L, let j=j+1 and continue the iteration starting from Eq. 11; otherwise, stop the iteration and compute the PLS transform12Display Formula

17WPLST=G(PTG)1(TTT)1TTRT,
based on which reflectance can be estimated as r̂=WPLSũ.

As L controls the number of iterations, its value is influential to PLS. If L=min(J,K), E and F become zeros and PLS reduces to OLS; otherwise, if L<min(J,K), the colinearity of matrix Ũ is reduced.12 In this work, L is determined according to the spectral accuracy of the reflectance estimation, as is discussed in the following section.

In the multispectral imaging system, we used a monochrome digital camera (model Cool-SNAP HQ2, Roper Scientific Incorporated, Ottobrunn, Germany) with 14-bit digitization and 16 narrowband filters (10nm half-width, product of Andover Company, Salem, New Hampshire) that uniformly cover the visible spectrum ranging from 400to700nm. The response of the camera deviates from linearity by less than 1%.14 As it is difficult to accurately acquire OECF at such a nonlinear level by imaging a number of gray samples, we consider it appropriate to treat it by extending the polynomial camera responses, as discussed in previous sections. We used 414 textile Pantone patches as the color targets, with half for training and half for testing. The surfaces of these patches contain weak textures and some gloss reflection, and hence are not ideal diffusers. The reflectance data of these patches were measured by the spectrophotometer GretagMacBeth (Grand Rapids, Michigan) 7000A; the 16-channel multispectral images were acquired by the imaging system. To reduce imaging noise, three sequential images were captured and averaged for the same scene, and the responses of each color sample were averaged in spatial areas with approximately 60×60pixels. The transform between camera responses and reflectance was calculated from the training set and evaluated on the test set. The estimation accuracy was examined by spectral root-mean-square (rms)1 error and CIEDE2000 color difference15 error under CIE standard illuminants. We tried different randomization strategies for selecting training and test samples, and found that the estimation accuracies were quite close. Hence we only present the experimental results of the case where the odd numbered samples were used for training and the rest for testing.

The performance of the PLS method is compared with the OLS and RLS methods. For the RLS method, we set the regularization parameter η=0.001, which approximately produces the minimum spectral rms error. For the PLS method, it is found that the spectral error is not very sensitive to the number of PLS component L. The suitable value of L is in the range from 40 to 60, and we adopted L=40 for computational efficiency. Table 1 gives the spectral and colorimetric errors of the PLS method, compared with the Wiener estimation, OLS, and RLS methods. It is clear that in terms of both colorimetric and spectral error metrics, the proposed method outperforms the Wiener estimation and OLS methods while being close to the RLS method. This is expected, as Wiener estimation cannot account for nonlinearity, and the OLS method has the inherent overfitting problem. The RLS method exhibits improved estimation accuracy through the introduction of the regularization term. By removing the common variance in the response and reflectance matrices, PLS also yields good accuracy.

Table Grahic Jump Location
Spectral rms errors and CIEDE2000 (ΔE00) errors of Wiener estimation, OLS, RLS, and PLS methods.

Due to the iterative nature of PLS, the computation time of the training procedure (calculation of WPLS) is approximately proportional to L. As the transform matrix WPLS is calculated before reflectance estimation, this computation time does not affect algorithm efficiency. The computation time of the test procedure is determined by the size of the matrix WPLS and is irrelevant to L. The PLS method was implemented under the MATLAB® environment and run on a PC with an Intel Core 2 CPU at 1.86GHz and with 3-GB memory. For a multispectral image with 1392×1040pixels, the PLS method costs about 9s and can be much faster if programmed using C language. This indicates that the computational efficiency of the PLS method is acceptable to many practical applications.

We propose a method for estimating reflectance from multichannel camera responses based on high-order polynomials and partial least squares. The proposed method is capable of dealing with nonlinearity in the imaging process. The appropriate number of PLS components is determined based on spectral rms error distribution. In terms of spectral and colorimetric error metrics, the proposed technique is superior to Wiener estimation and polynomial regression solved by ordinary least squares, and is close to polynomial regression solved by regularized least squares.

We thank the reviewers for their comments that substantially improved this work. This work was supported by the NSF of China under grant 60778050.

Shen  H. L., , Xin  J. H., , and Shao  S. J., “ Improved reflectance reconstruction for multispectral imaging by combining different techniques. ,” Opt. Express.  1094-4087 15, (9 ), 5531–5536  ((2007)).
DiCarlo  J. M., and Wandell  B. A., “ Spectral estimation theory: beyond linear but before Bayesian. ,” J. Opt. Soc. Am. A.  0740-3232 20, (7 ), 1261–1270  ((2003)).
Murakami  Y., , Obi  T., , Yamaguchi  M., , and Ohyama  N., “ Nonlinear estimation of spectral reflectance based on Gaussian mixture distribution for color image reproduction. ,” Appl. Opt..  0003-6935 41, (23 ), 4840–4847  ((2002)).
Shen  H. L., and Xin  J. H., “ Colorimetric and spectral characterization of a color scanner using local statistics. ,” J. Imaging Sci. Technol..  1062-3701 48, (4 ), 342–346  ((2004)).
Connah  D. R., and Hardeberg  J. Y., “ Spectral recovery using polynomial models. ,” in Proc. SPIE.  0277-786X 5667, , 65–75  ((2005)).
Heikkinen  V., , Jetsu  T., , Parkkinen  J., , Hauta-Kasari  M., , Jaaskelainen  T., , and Lee  S. D., “ Regularized learning framework in the estimation of reflectance spectra from camera responses. ,” J. Opt. Soc. Am. A.  0740-3232 24, (9 ), 2673–2683  ((2007)).
Shimano  N., “ Recovery of spectral reflectances of objects being imaged without prior knowledge. ,” IEEE Trans. Image Process..  1057-7149 15, (7 ), 1848–1856  ((2006)).
Pratt  W. K.,  Digital Imaging Processing. , 2nd ed.,  Wiley ,  New York  ((1991)).
Shen  H. L., , Xin  J. H., , Yang  D. X., , and Lou  D. W., “ Estimation of optoelectronic conversion functions of imaging devices without using gray samples. ,” Color Res. Appl..  0361-2317 33, (2 ), 135–141  ((2008)).
Hong  G., , Luo  M. R., , and Rhodes  P. A., “ A study of digital camera colorimetric characterization based on polynomial modeling. ,” Color Res. Appl..  0361-2317 26, (1 ), 76–84  ((2001)).
Geladi  P., and Kowalski  B. R., “ Partial least-squares regression: a tutorial. ,” Anal. Chim. Acta.  0003-2670 185, , 1–7  ((1986)).
Chiang  L. H., , Russell  E. L., , and Braatz  R. D.,  Fault Detection and Diagnosis in Industrial Systems. ,  Springer ,  London  ((2001)).
Press  W. H., , Teukolsky  S. A., , Vetterling  W. T., , and Flannery  B. P.,  Numerical Recipes in C—the Art of Scientific Computing. , 2d ed.,  Cambridge ,  UK  ((1992)).
Luo  M. R., , Cui  G., , and Rigg  R., “ The development of the CIE 2000 colour-difference formula: CIEDE2000. ,” Color Res. Appl..  0361-2317 26, (5 ), 340–350  ((2001)).
© 2010 SPIE and IS&T

Citation

Hui-Liang Shen ; Hui-Jiang Wan and Zhe-Chao Zhang
"Estimating reflectance from multispectral camera responses based on partial least-squares regression", J. Electron. Imaging. 19(2), 020501 (April 12, 2010). ; http://dx.doi.org/10.1117/1.3385782


Figures

Tables

Table Grahic Jump Location
Spectral rms errors and CIEDE2000 (ΔE00) errors of Wiener estimation, OLS, RLS, and PLS methods.

References

Shen  H. L., , Xin  J. H., , and Shao  S. J., “ Improved reflectance reconstruction for multispectral imaging by combining different techniques. ,” Opt. Express.  1094-4087 15, (9 ), 5531–5536  ((2007)).
DiCarlo  J. M., and Wandell  B. A., “ Spectral estimation theory: beyond linear but before Bayesian. ,” J. Opt. Soc. Am. A.  0740-3232 20, (7 ), 1261–1270  ((2003)).
Murakami  Y., , Obi  T., , Yamaguchi  M., , and Ohyama  N., “ Nonlinear estimation of spectral reflectance based on Gaussian mixture distribution for color image reproduction. ,” Appl. Opt..  0003-6935 41, (23 ), 4840–4847  ((2002)).
Shen  H. L., and Xin  J. H., “ Colorimetric and spectral characterization of a color scanner using local statistics. ,” J. Imaging Sci. Technol..  1062-3701 48, (4 ), 342–346  ((2004)).
Connah  D. R., and Hardeberg  J. Y., “ Spectral recovery using polynomial models. ,” in Proc. SPIE.  0277-786X 5667, , 65–75  ((2005)).
Heikkinen  V., , Jetsu  T., , Parkkinen  J., , Hauta-Kasari  M., , Jaaskelainen  T., , and Lee  S. D., “ Regularized learning framework in the estimation of reflectance spectra from camera responses. ,” J. Opt. Soc. Am. A.  0740-3232 24, (9 ), 2673–2683  ((2007)).
Shimano  N., “ Recovery of spectral reflectances of objects being imaged without prior knowledge. ,” IEEE Trans. Image Process..  1057-7149 15, (7 ), 1848–1856  ((2006)).
Pratt  W. K.,  Digital Imaging Processing. , 2nd ed.,  Wiley ,  New York  ((1991)).
Shen  H. L., , Xin  J. H., , Yang  D. X., , and Lou  D. W., “ Estimation of optoelectronic conversion functions of imaging devices without using gray samples. ,” Color Res. Appl..  0361-2317 33, (2 ), 135–141  ((2008)).
Hong  G., , Luo  M. R., , and Rhodes  P. A., “ A study of digital camera colorimetric characterization based on polynomial modeling. ,” Color Res. Appl..  0361-2317 26, (1 ), 76–84  ((2001)).
Geladi  P., and Kowalski  B. R., “ Partial least-squares regression: a tutorial. ,” Anal. Chim. Acta.  0003-2670 185, , 1–7  ((1986)).
Chiang  L. H., , Russell  E. L., , and Braatz  R. D.,  Fault Detection and Diagnosis in Industrial Systems. ,  Springer ,  London  ((2001)).
Press  W. H., , Teukolsky  S. A., , Vetterling  W. T., , and Flannery  B. P.,  Numerical Recipes in C—the Art of Scientific Computing. , 2d ed.,  Cambridge ,  UK  ((1992)).
Luo  M. R., , Cui  G., , and Rigg  R., “ The development of the CIE 2000 colour-difference formula: CIEDE2000. ,” Color Res. Appl..  0361-2317 26, (5 ), 340–350  ((2001)).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Related Book Chapters

Topic Collections

Advertisement
  • Don't have an account?
  • Subscribe to the SPIE Digital Library
  • Create a FREE account to sign up for Digital Library content alerts and gain access to institutional subscriptions remotely.
Access This Article
Sign in or Create a personal account to Buy this article ($20 for members, $25 for non-members).
Access This Proceeding
Sign in or Create a personal account to Buy this article ($15 for members, $18 for non-members).
Access This Chapter

Access to SPIE eBooks is limited to subscribing institutions and is not available as part of a personal subscription. Print or electronic versions of individual SPIE books may be purchased via SPIE.org.