Regular Articles

Hyperspectral imaging using RGB color for foodborne pathogen detection

[+] Author Affiliations
Seung-Chul Yoon, Tae-Sung Shin, Kurt C. Lawrence, Gerald W. Heitschmidt, Bosoon Park, Gary R. Gamble

U.S. National Poultry Research Center, U.S. Department of Agriculture-Agricultural Research Service, 950 College Station Road, Athens, Georgia 30605, United States

J. Electron. Imaging. 24(4), 043008 (Aug 10, 2015). doi:10.1117/1.JEI.24.4.043008
History: Received April 1, 2015; Accepted July 2, 2015
Text Size: A A A

Open Access Open Access

Abstract.  This paper reports the development of a spectral reconstruction technique for predicting hyperspectral images from RGB color images and classifying food-borne pathogens in agar plates using reconstructed hyperspectral images. The six representative non-O157 Shiga-toxin producing Escherichia coli (STEC) serogroups (O26, O45, O103, O111, O121, and O145) grown on Rainbow agar plates were used for the study. A line-scan pushbroom hyperspectral imaging spectrometer was used to scan full reflectance spectra of pure non-O157 STEC cultures in the visible and near-infrared spectral range from 400 to 1000 nm. RGB color images were generated by simulation from hyperspectral images. Polynomial multivariate least-squares regression analysis was used to reconstruct hyperspectral images from RGB color images. The mean R-squared value for hyperspectral image reconstruction was 0.98 in the spectral range between 400 and 700 nm for linear, quadratic, and cubic polynomial regression models. The accuracy of the hyperspectral image classification algorithm based on k-nearest neighbors algorithm of principal component scores was validated to be 92% with the test set (99% with the original hyperspectral images). The results of the study suggested that color-based hyperspectral imaging would be feasible without much loss of prediction accuracy compared to true hyperspectral imaging.

Figures in this Article

Detection and identification of foodborne pathogens are increasingly important for development of intervention and verification strategies for the food industry and regulatory agencies. Traditional culture-based direct plating methods are still the gold standard for presumptive-positive pathogen screening in many microbiology laboratories, where agar media are routinely used for isolation, enumeration, and detection of pathogenic bacteria. In practice, highly skilled technicians visually screen and manually select presumptive-positive colonies by trial and error for microscopic, biochemical, serological, and molecular confirmation tests whose results may or may not be obtained rapidly. Therefore, the culture methods are labor intensive and prone to human subjective errors. Another challenge with direct plating is that competitive microflora often grow together with target microorganisms on agar media and can appear morphologically similar.

Hyperspectral image classification algorithms in the visible and near-infrared (VNIR) spectral range from 400 to 1000 nm have been previously developed for automated screening of pathogen colonies on agar plates, which included Campylobacter, Salmonella, and Shiga toxin-producing Escherichia coli (STEC).15 The key idea of using hyperspectral imaging was to find spectral and spatial features unique to the bacterial colonies on agar and utilize the spectral and/or spatial features for detection and classification so that relatively simple classification algorithms, such as k-nearest neighbor (kNN) and quadratic discriminant analysis, could be applied. Although hyperspectral imaging using full spectra demonstrated the good sensitivity and specificity to detect pathogen colonies and identify their types, the high cost of a hyperspectral imaging system hinders its routine use as a screening tool in microbiology laboratories. RGB color (or multispectral) imaging techniques can be a cost-effective solution, although a commercial multispectral imaging solution is still much more expensive than color imaging. In addition, either a color-based or multispectral imaging solution for pathogen detection, such as non-O157 STECs, requires the development of a new classification algorithm. Therefore, an idea was to use the already developed hyperspectral image classification algorithm(s) while using color (or multispectral) imaging.

The use of multispectral imaging to recover reflectance spectra has been extensively studied in the color science field typically with color charts or targets with known spectra.6 There are many techniques to recover reflectance spectra from multispectral (typically color) information.6,7 Some of the hyperspectral estimation techniques to recover reflectance spectra include Wiener estimation, multiple regression analysis, Maloney-Wandell method, Imai-Berns method, and Shi-Healey method.612 Hyperspectral images have also been used to develop a hyperspectral estimation method from color and/or multispectral images for spectral estimation of paint.8 A different group of techniques for hyperspectral estimation is based on sparse signal representation and sampling, such as compressive sensing. Sparse representation methods were developed to increase the spatial resolution of hyperspectral images with spectral unmixing and L1 minimization.13,14 Compressive sensing techniques were applied to reconstruct hyperspectral images from undersampled fluorescent microscopic images in biomedical imaging15 and from a few noisy multispectral images in remote sensing.16 Active spectral illumination has been studied to quantify complex spectral responses of tissue oxygenation with sparse sampled multispectral images.17 But the use of color to recover hyperspectral images is still an underexplored research area, especially to the hyperspectral imaging researchers in the field of agriculture and food science and engineering.

In this paper, polynomial multivariate regression analysis was used to recover reflectance spectra from RGB color images and to reconstruct full VNIR hyperspectral images that would be applied to a hyperspectral image classification algorithm specifically developed for detection of non-O157 STEC colonies on agar plates. Thus, the overall objective of the study was to investigate the potential of using color to recover reflectance spectra in the form of hyperspectral images and to evaluate the performance of the hyperspectral image classification algorithm with the recovered hyperspectral images.

Pathogenic Bacteria

The pathogenic bacterial cultures were obtained from a culture collection at the Eastern Laboratory of U.S. Department of Agriculture–Food Safety and Inspection Service (FSIS) according to the FSIS protocol.18 A total of six non-O157 STEC strains were chosen from each of the big six O-serogroups (O26, O45, O103, O111, O121, and O145). The details on how to prepare the non-O157 STEC bacteria are provided in our previous publications.35 Three replicated experiments were carried out to grow the bacterial colonies on Rainbow agar plates. Each experiment generated two sets of six agar plates inoculated with two cell concentrations of the pure cultures, where only one type of bacteria was grown in an agar plate. Serial dilutions of each cell suspension were prepared in 50 and 100 colony forming units (CFU) (50 and 100μL aliquots of 103CFUmL1 dilutions) that were inoculated onto Rainbow agar plates by a spread plating technique. All inoculated plates were incubated at 37°C for 24 h. A total of 36 agar plates (2dilutions×3replicates×6serogroups) were prepared.

Hyperspectral Images

Hyperspectral image acquisition was done with a pushbroom line-scan hyperspectral imaging system (Themis Vision Systems, Richmond, Virginia) in the VNIR spectral range from 400 to 1000 nm. The imaging system, shown in Fig. 1, consisted of a 12-bit CCD camera with 1376×1040pixels (SensiCamQE, The Cooke Corporation, Romulus, Michigan), an imaging spectrograph (ImSpector V10E with 30μm slit, Specim-Spectral Imaging Ltd., Oulu, Finland), a C-mount objective lens (APO-Xenoplan 1.8/35mm, Schneider Optics), a motion control (Newark, California), a custom sample-holder, and a computer. The front objective lens was moved to scan one spatial line for one scan such that the dispersed full spectra along the line were recorded on the CCD detector. Further details on the system are provided in our previous publications.35

Graphic Jump Location
Fig. 1
F1 :

Hyperspectral imaging system.

The average wavelength interval was 1.27 nm. The dimensions of a raw hyperspectral data cube after 2 (spatial) by 2 (spectral) hardware binning on camera were 688(W)×500(H)×520(wavelengths) from 380 to 1000 nm. Measured reflectance values were calibrated (i.e., normalized) to relative reflectance R with a 40% reflectance Spectralon® target (13×13cm, SRT-75-050, Labsphere, North Sutton, New Hampshire), as described in Refs. 7 and 8. The spectral dimension of each image was reduced to 473 spectral bands ranging from 400 to 1000 nm by removing extreme wavelength bands. Thus, the resulting dimension of each hyperspectral data cube became 688(W)×500(H)×473(wavelengths). Also, spectral noise was reduced by a Savitzky-Golay smoothing filter (window size: 25; order of moment: 4) at each pixel position. After that, the calibrated hyperspectral images (relative reflectance) were stitched together into a single image mosaic. The images from the same dilution were added to each column of the mosaic from left (less cells) to right (more cells). Figure 2 shows two color-composite image mosaics (enhanced by 2% linear stretch for display purpose only) obtained from 36 calibrated hyperspectral reflectance images divided into training and test sets with 18 images for each set. The dimensions of each mosaic were 2064×3000×473pixels with floating-point values. The file size of each mosaic was 11.7GB. The 18 hyperspectral images in each mosaic were arranged in six rows and three columns, where rows consisted of images with pure cultures of O26, O45, O103, O111, O121, and O145 from top to bottom, and columns referred to replicates and different cell dilutions.

Graphic Jump Location
Fig. 2
F2 :

Hyperspectral reflectance image mosaics (color-composites): (a) training set and (b) test set. The hyperspectral images of each of six serogroups were arranged into one row from top to bottom. The images in each column were the cultures of the same cell dilution, as denoted by the target colony forming unit (CFU).

RGB Color Images

The color images (called camera responses) were obtained from the calibrated hyperspectral images by simulation. Mathematically, the camera responses ci at each color channel and at each image pixel location (x,y) were calculated by the following: Display Formula

ci(x,y)=λ=4001000r(x,y,λ)β(λi),i=1,2,3,(1)
where r(x,y,λ) and βi(λ) are a reflectance spectrum at (x,y) and the spectral sensitivity of the camera’s i’th channel, respectively. A total of 473 wavelengths, λ’s, were defined at nonuniform intervals from 400 to 1000 nm, whose average interval was 1.27 as previously mentioned. The spectral sensitivities were resampled at the same wavelengths as the hyperspectral data.

Three spectral sensitivity functions were compared in order to determine the best method to extract color images from hyperspectral images. The first method, assuming the synthetic unit-impulse functions for a camera’s spectral sensitivities, was to use three spectral band images obtained at three discrete wavelengths, 455, 535, and 610 nm for blue, green, and red color channels, respectively. The second method, assuming the synthetic rectangular functions for the sensitivities, was to use three spectral ranges of nonoverlapping wavelengths at 435 to 485 (blue), 505 to 575 (green), and 595 to 660 (red) nm. Average reflectance values computed in each wavelength range were assigned to the corresponding RGB values. The third method, which was more practical, was to use the spectral sensitivities of a CCD sensor (ICX445AQ, Sony Corp.) used in a commercial RGB color camera (Prosilica GC1290C, Allied Vision Technologies GmbH, Stadtroda, Germany), whose spectral sensitivities are shown in Fig. 3. The RGB values computed by Eq. (1) were converted to unsigned 16 bits.

Graphic Jump Location
Fig. 3
F3 :

Spectral sensitivities of a color CCD sensor (Sony ICX445AQ) on Prosilica GC1290C.

Region of Interest

Both hyperspectral and RGB images of agar plates were masked by binary plate masks in Fig. 4 in order to suppress the unwanted background outside of each agar plate. The plate masks were created manually by a drawing tool in the ENVI software (Exelis Visual Information Solutions, Boulder, Colorado).

Graphic Jump Location
Fig. 4
F4 :

Plate binary mask mosaics of (a) training set and (b) test set.

The ground-truth colony regions of interest (ROIs) representing the identity (i.e., STEC serogroup) of each colony were obtained by an automatic colony segmentation algorithm utilizing the characteristics of colony morphology and response to lights.19 As shown in Fig. 5, the colony segmentation images were stitched into two mosaics for training and testing in the same way as the hyperspectral image mosaics. The choice of the ROIs was studied to determine whether there is a sampling bias caused by different ROI (i.e., sampling).

Graphic Jump Location
Fig. 5
F5 :

Region-of-interest (ROI) image mosaics of (a) training set and (b) test set.

Seven different sample sets derived from the original colony ROIs and agar background regions in the training set were compared to find the best sampling size for hyperspectral data reconstruction from RGB color data. Figure 6 shows the examples of the defined ROI sets. Each sample set consisted of (1) all pixels in each plate image (excluding the outside of agar plate) defined by a binary plate mask (MASK), (2) randomly selected 5% of total pixels in each plate mask (MASK5%), (3) all pixels in each colony ROI plus randomly selected agar pixels (COLONY), (4) four neighboring pixels from the center position of each colony ROI plus the center position pixel (a total of five colony pixels) and randomly selected agar pixels (COLONY5p), (5) one mean spectrum from each colony ROI plus randomly selected agar pixels (COLONYa), and finally (6) all pixels in morphologically dilated (i.e., spatially extended) colony ROIs and randomly selected agar pixels (COLONYd). The number of randomly selected agar pixels from (3) to (6) was 30% of the total number of pixels in each ROI set, which was heuristically determined.

Graphic Jump Location
Fig. 6
F6 :

ROI examples are (a) plate mask ROI (MASK), (b) 5% random samples (MASK5%), (c) colony ROIs plus random pixels (COLONY), (d) 5-colony pixels plus random pixels (COLONY5p), (e) mean spectra (denoted at the center of each colony in the image) of each colony ROI plus random pixels (COLONYa), and (f) enlarged colony ROIs and random pixels (COLONYd).

Reconstruction of Hyperspectral Data

Given the reflectance spectra and RGB values of n observations, which were obtained from the n pixels on the previously mentioned ROIs, the hyperspectral data reconstruction problem was formulated as a system of linear equations as follows: Display Formula

R=C·T,(2)
where R, C, and T are a hyperspectral reflectance data matrix of n row vectors each with 473 reflectance values, a camera response matrix of n row vectors each with an RGB color 3-tuple, and the transform matrix (unknown) to be estimated, respectively. There are several solutions available in the literature.612 In this paper, polynomial multivariate least-squares regression (PMLR) was used to estimate the transform matrix T and thus the hyperspectral data. The polynomial regression model assumes that there is a nonlinear relationship between the predictors (color image) and the dependent variables (hyperspectral image). The degree of a polynomial is an unknown control parameter that determines the behavior of the independent variables (predictors) to fit to the hyperspectral data. The optimal degree was determined in a heuristic way, for which the regression performance was compared. A total of five PMLR models from first- to fifth-degree polynomials was compared to find the best estimation method. The camera response matrix C in Eq. (2) was expanded to include higher-order polynomial terms as follows: Display Formula
R=Ce·T,(3)
where Ce=[c1;;cn] is an expanded camera response matrix of n row vectors ci, i=1,2,,n. For example, a row vector ci consists of [c1,c2,c3,c1×c1,c2×c2,c3×c3,c1×c2,c1×c3,c2×c3,,] higher-order terms. The Moore–Penrose pseudoinverse method was used to solve Eq. (3) and to obtain the estimated transform matrix T^ as follows: Display Formula
T^=Ce+·R.(4)
Then, given the RGB color values, the hyperspectral data were estimated by the transform matrix by plugging into Eq. (3).

The goodness of fit (GOF) of hyperspectral reconstruction was measured by the coefficient of determination (R2) and the root mean square deviation (RMSD, also called root mean square error), which were defined as follows: Display Formula

R2=1k=1Nλ(rkr^k)2k=1Nλ(rkμr)2,μr=k=1NλrkNλ,RMSD=k=1Nλ(rkr^k)2Nλ,(5)
where rk and r^k are the measured and estimated reflectance values at the k’th wavelength band, respectively. The number of wavelength points was Nλ (N=473). The R2 value is a statistical measure of how well the predicted data agree with the measured data. An R2 of 1 refers to a perfect fit of a regression line to the measured data. The RMSD represents the sample standard deviation of the differences between predicted values and measured values.

Hyperspectral Image Classification Algorithm

A hyperspectral image classification algorithm for non-O157 STEC colony classification was based on chemometric preprocessing of spectral data, including the transformation of measured reflectance spectra to absorbance spectra, standard normal variate and detrending for correction of light scattering, first derivative, and moving average smoothing.35 Then, the preprocessed spectra were transformed by principal component analysis. The classification for the six STEC classes was performed by the kNN algorithm (with k=3) of scores in the principal component subspace spanned by six principal components.35 Note that the reflectance spectra at each pixel were transformed to absorbance log10(1/R) for classification, whereas the prediction of hyperspectral data from color data was done with reflectance spectra.

When extracting spectral data from pixel locations defined by the various ROIs, the data were unfolded into an n×473 spectral data matrix X whose values were associated with n observations (number of samples in pixels) in rows and 473 independent variables (wavelengths) in columns. A response vector of class labels from 1 to 6 was also created. The training set for classification consisted of 916 ROIs (i.e., colonies) with 63,136 pixels (i.e., observations or samples), whereas the test set for validation of classification models consisted of 1250 colonies with 77,473 pixels. The classification model was trained with both original and reconstructed hyperspectral data and validated with the training and test sets. All spectral data were treated in the same way as the aforementioned data preprocessing techniques and unfolded into a data matrix. The final decision-making rule was applied at colony level by the winner-take-all strategy (simple majority voting) of prediction results at pixel level. Prediction results of each sample in the training or test set were projected back onto the corresponding image domain to create a hyperspectral image.

Spectral Sensitivity Functions

The RGB color images obtained by simulation of a camera sensitivity function (Sony ICX445AQ CCD) are shown in Fig. 7. The other RGB images are not shown because they were visually similar to each other. The displayed color images in Fig. 7 were enhanced by 2% linear stretch for display purpose.

Graphic Jump Location
Fig. 7
F7 :

RGB color images simulated by a camera sensitivity function (Sony ICX445AQ CCD): (a) training set and (b) test set.

The GOF of the regression models in relation to spectral sensitivity functions is summarized in Table 1.

Table Grahic Jump Location
Table 1Goodness of fit001 of regression models in relation to spectral sensitivity functions.
Table Footer NoteaAverage values over 400 to 1000 nm. The number in () is the average over 400 to 700 nm.

The average R2 and RMSD values in Table 1 were obtained from the validation of the regression models with all samples in the test set. The statistical significance of R2 and RMSD values varying among different spectral sensitivity functions was tested by the analysis of variance (ANOVA). The p values resulting from the ANOVA test were 0.33 (p>0.05) and 0.34 (p>0.05) for R2 and RMSD, respectively. Hence, the selection of a particular spectral sensitivity function would not be statistically significant. Nonetheless, when the three different spectral sensitivity functions were compared, the spectral sensitivity functions emulating them of a commercial color camera with the broad range of wavelengths produced the best fit spectra, especially over the spectral range between 400 and 700 nm. Therefore, the spectral sensitivity functions of a real camera covering the broad spectral ranges were used for the development of the regression models later on.

In this study, there was no need to solve an image registration problem to find the corresponding points or regions between the RGB and hyperspectral images, because RGB color values used to compute the transform matrix were created by the simulation of three different camera sensitivity functions from the defined ROI sample locations on hyperspectral images. However, in real-life applications of hyperspectral estimation from color, it would be important to calibrate a color camera with measured camera responses (color values) and their corresponding spectra measured by a hyperspectral image sensor, which will be a topic for our future research.

Sample Sizes

The size of samples (i.e., the number of pixels) of each ROI set in the training set is summarized in the first row of Table 2. The sample sizes selected from the training set with the six different ROI sets were evaluated and compared on the basis of the regression performance with the test set. The ANOVA test of the GOF values (R2 and RMSD), similar to the comparison of the spectral sensitivity functions, showed that the differences in sample size were statistically significant (p<0.001). Therefore, it was important to find the best sample size for the regression models. The GOFs of each different sample size are summarized in Table 2. The average R2 and RMSD values in Table 2 were computed with the samples (5% random selection, ROI set MASK5%) in the test set and for all regression models.

Table Grahic Jump Location
Table 2Goodness of fits002 of regression models in relation to sample sizes.
Table Footer NoteaAverage values over 400 to 1000 nm. The number in () is the average over 400 to 700 nm.

The average R2 values tended to increase as sample size increased. The 5% random selection of pixels in the training set for training the regression models showed the best prediction (or called regression) performance, closely followed by all pixels in plate masks. The RMSD values <1 could mean <1% difference in percent reflectance. Note that this small difference in reflectance did not mean uniform performance regardless of regression model, sample location, material type (serogroup), and wavelength.

Hyperspectral Image Reconstruction

The prediction performance of the five regression models was evaluated by the fitting accuracy of the reconstructed hyperspectral data to the measured hyperspectral data in terms of GOFs. Figure 8 shows the average R2 and RMSD values of the five regression models, where each model was trained with all samples (ROI set MASK, N=3,058,576) in the training set and tested with all samples (ROI set MASK, N=3,121,509) in the test set. Overall, linear, quadratic, and cubic polynomial regression models showed better performance than the higher-order polynomial regression models.

Graphic Jump Location
Fig. 8
F8 :

Prediction performance of regression models. p is a polynomial order. With the test set, (a) the average R2 values were computed for all test samples over all wavelengths, and (b) the average root mean square deviation values were computed for all test samples at each wavelength.

Figure 8(b) shows that the spectral data in the spectral range below 700 nm would be better predicted by all five regression models than the spectral range above 700 nm, although the quintic (p=5) polynomial model was the worst. The prediction performance of the regression models measured by α, not shown here, was also similar to the performance measured by R2 and RMSD. The cubic (p=3) polynomial regression model showed the least prediction errors in terms of the GOFs when measured with all samples in the test set.

The prediction performance of the regression models was also evaluated in relation to colony identities (STEC serogroups in our case). Figure 9 shows the mean reflectance spectra obtained from the colony ROIs on the original and the regressed hyperspectral images, when the regression models were trained with all pixels on the plate masks (ROI set MASK, N=3,058,576) in the training set and tested with all colony ROIs (1250 colonies with 77,473 pixels) in the test set.

Graphic Jump Location
Fig. 9
F9 :

Mean reflectance spectra obtained by the regression models for each Shiga toxin-producing Escherichia coli (STEC) serogroup [(a) O26, (b) O45, (c) O103, (d) O111, (e) O121, and (f) O145)]: trained with all pixels on plate masks (ROI set MASK) in the training image set and validated on colonies in the test set.

The analysis of the mean reflectance spectra showed overall better fit of all regression models in the wavelengths below 700 nm. Large variations in mean reflectance were observed at wavelengths longer than 700 nm. The cause of the large variations in the longer wavelength region was not studied in this paper. Thus, we only speculated that first, the simulated RGB values did not contain the spectral information in the wavelength and/or second, the variation in the original data was too large to model with the regression models. Nonetheless, the first three-order polynomial regression models were better than the higher-order polynomial models. The linear regression model showed the best prediction performance for O26 and O103, whereas the quadratic (p=2) polynomial regression model was the best (or close to the best) for O45, O111, and O121, and the cubic (p=3) model was the best for O121 and O145.

Figure 10 shows the mean reflectance spectra obtained from the colony ROIs on the original and the regressed hyperspectral images, when the regression models were trained with samples corresponding to the colony ROIs and background agar (ROI set COLONY) in the training set and tested with all colony ROIs (1250 colonies with 77,473 pixels) in the test set. As shown in Fig. 10, when the regression models were trained with the ROI set COLONY (colony ROIs plus random agar pixels), the overall prediction performance was greatly improved over all wavelengths. Especially, the large variation at the longer wavelengths seen in Fig. 9 was reduced. This experiment confirmed that the camera sensitivity functions were indeed not statistically significant, but the difference in same size about how to sample the data in the training set was statistically significant.

Graphic Jump Location
Fig. 10
F10 :

Mean reflectance spectra obtained by the regression models for each STEC serogroup [(a) O26, (b) O45, (c) O103, (d) O111, (e) O121, and (f) O145)]: trained with colony ROIs and some agar samples (ROI set COLONY) in the training set and validated on colonies in the test set.

The linear regression model showed the best (or close to the best) prediction performance in terms of spectral analysis for all serogroups except O111. Although sampling only with the ROI set COLONY showed better regression performance than sampling with the entire population in the training set, the performance of the developed classification model may or may not be directly correlated with these findings because the classification models including image segmentation and chemometric preprocessing were optimized with the original hyperspectral images.

Figure 11(a) shows RGB color-composite images obtained from the reconstructed hyperspectral images, and Fig. 11(b) shows pseudo-color images derived from the normalized sum of differences between the measured and predicted hyperspectral images over all wavelengths, both using the linear regression model trained with all pixels in the plate masks. The normalized sums of reflectance differences at each image pixel were mapped to a color according to a scale between 0 (small error: blue at the bottom of the color bar) and 1 (large error: red at the top of the color bar). The regression errors were more prominent at the colonies. The centers of the colonies tended to produce more regression model errors. From Fig. 11(b), it was implied that the spectra of the O145 colonies were the most difficult to predict using the regression model.

Graphic Jump Location
Fig. 11
F11 :

(a) Color-composite images obtained from reconstructed hyperspectral images and (b) pseudo-color images derived from the normalized sum of differences between the measured and predicted hyperspectral images over all wavelengths.

STEC Classification Results

The performance of the pathogen classification algorithm was measured in terms of overall classification accuracy. The performance of the classification algorithm depended greatly on the characteristics of the samples to train, which was similar to the case of the regression models mentioned in this paper. Due to this issue, three different ROI sets, including MASK (all pixels on plate masks), COLONYa (mean spectra plus random agar pixels), and COLONY5p (five pixels near the colony center), were compared for the performance evaluation of the classifier. The linear regression model was used for the reconstruction of hyperspectral images and trained with the ROI set MASK (all samples) in the training set. The linear regression model was applied to predict (i.e., reconstruct) hyperspectral images from the color images in both training and test sets. The classification algorithm was retrained with the reconstructed hyperspectral images in the training set. For validation, the classification algorithm was applied to all pixels of the ROI set MASK in both original and reconstructed hyperspectral images and validated on colony ROIs for performance assessment. Table 3 shows the overall classification accuracy of the classification algorithm trained with three different sets of samples in the training dataset.

Table Grahic Jump Location
Table 3Overall classification accuracy of classification models observed at the colony level.

The training accuracy varied from 66 to 99%, where the best performance was achieved by the classification model trained with all pixels in the colony ROIs (99%), whereas the best test performance was achieved by the model trained with mean colony spectra (92.3%). When compared with the results from the original hyperspectral images, the classification accuracy based on the reconstructed hyperspectral images varied more greatly than the case of the original images. One reason is because the classification algorithm was not optimized with the reconstructed hyperspectral images. It is known that there are very subtle spectral differences among different STEC classes. Therefore, any small error in reconstruction may have caused unexpected large errors and variations in classification results. Nonetheless, the reason for this large variation in performance needs to be studied in the future. Also, the classification algorithm developed with the original hyperspectral images may not be optimal for the reconstructed hyperspectral images. Therefore, a new hyperspectral image classification algorithm for STEC detection may need to be developed in the context of different regression models in the future.

This study showed a potential of the use of RGB color to reconstruct hyperspectral data and its application to classification of non-O157 STEC colonies on agar media using a hyperspectral image classification algorithm. Contrary to the most previous work using color charts or targets with known spectra, the current work used the hyperspectral images of the actual targets (microbial colonies on agar media) to obtain the spectra of the real targets of interest for reconstructing the spectra from color images by polynomial multivariate least-squares regression. A limitation of the current study was the use of simulated color images for reconstructing hyperspectral images by the regression analysis. Polynomial multivariate least-squares regression models using up to the fifth-order polynomial were evaluated in relation to the camera’s spectral sensitivity functions and sample sizes. The regression performance measured by GOF measures (coefficient of determination and RMSD) was similar among the linear, quadratic, and cubic polynomial regression models. The effect of different spectral sensitivities on the regression performance was not statistically significant, but that of different sample sizes was statistically significant. The spectra in the wavelengths shorter than 700 nm were better predicted by the regression models, but the spectra at the wavelengths longer than 700 nm were not well predicted. The analysis of mean reflectance spectra revealed the potential of linear, quadratic, and cubic regression models. The classification algorithm trained with the reconstructed hyperspectral images achieved up to 92% classification accuracy, whereas the same algorithm with the original hyperspectral images achieved 99% classification accuracy. The classification results indicated that the GOF measures were not directly matched with the classification accuracy, which needs to be further investigated later. In conclusion, although there were mixing results and still unanswered questions, there is still room for improvement of hyperspectral image regression and classification accuracy for non-O157 STEC detection using a color camera.

The authors would like to thank Jerrie Barnett and Peggy Feldner in the Quality and Safety Assessment Research Unit in Athens for their assistance in this research.

Yoon  S. C.  et al., “Hyperspectral reflectance imaging for detecting a foodborne pathogen: Campylobacter,” Trans. ASABE. 52, (2 ), 651 –662 (2009).CrossRef
Yoon  S. C.  et al., “Detection of Campylobacter colonies using hyperspectral imaging,” Sens. Instrum. Food Qual. Saf.. 4, (1 ), 35 –49 (2010).CrossRef
Yoon  S. C.  et al., “Hyperspectral imaging for differentiating colonies of non-O157 Shiga-toxin producing Escherichia coli (STEC) serogroups on spread plates of pure cultures,” J. Near Infrared Spectrosc.. 21, (2 ), 81 –95 (2013).CrossRef
Windham  W. R.  et al., “Hyperspectral imaging of Shiga toxin-producing Escherichia coli serogroups O26, O45, O103, O111, O121, and O145 on Rainbow agar,” J. Food Prot.. 76, (7 ), 1129 –1136 (2013). 0362-028X CrossRef
Yoon  S. C.  et al., “Differentiation of big-six non-O157 Shiga-toxin producing Escherichia coli (STEC) on spread plates of mixed cultures using hyperspectral imaging,” Food Meas.. 7, (2 ), 47 –59 (2013).CrossRef
Westland  S., , Ripamonti  C., and Cheung  V., Computational Colour Science Using MATLAB. , 2nd ed.,  Wiley ,  Hoboken, New Jersey  (2012).
López-Alvarez  M. A.  et al., “Selecting algorithms, sensors, and linear bases for optimum spectral recovery of skylight,” J. Opt. Soc. Am. A. 24, (4 ), 942 –956 (2007).CrossRef
Tsumura  N., , Miyake  Y., and Bochko  V., “Spectral color imaging system for estimating spectral reflectance of paint,” J. Imaging Sci. Technol.. 51, (1 ), 70 –78 (2007). 1062-3701 CrossRef
Haneishi  H.  et al., “System design for accurately estimating the spectral reflectance of art paintings,” Appl. Opt.. 39, (35 ), 6621 –6632 (2000). 0003-6935 CrossRef
Stigell  P., , Miyata  K., and Hauta-Kasari  M., “Wiener estimation method in estimating of spectral reflectance from RGB images,” Pattern Recognit. Image Anal.. 17, (2 ), 233 –242 (2007). 1054-6618 CrossRef
Imai  F. H., and Berns  R. S., “Spectral estimation using trichromatic digital cameras,” in  Proc. of the Int. Symp. on Multispectral Imaging and Color Reproduction for Digital Archives ,  Chiba, Japan , pp. 42 –49 (1999).
Valero  E. M.  et al., “Recovering spectral data from natural scenes with an RGB digital camera and colored filters,” Color Res. Appl.. 32, (5 ), 352 –360 (2007).CrossRef
Kawakami  R.  et al., “High-resolution hyperspectral imaging via matrix factorization,” in  IEEE Conf. on Computer Vision and Pattern Recognition , pp. 2329 –2336 (2011).
Guo  Z., , Wittman  T., and Osher  S., “L1 unmixing and its application to hyperspectral image enhancement,” Proc. SPIE. 7334, , 73341M  (2009). 0277-786X CrossRef
Studer  V.  et al., “Compressive fluorescence microscopy for biological and hyperspectral imaging,” Proc. Natl. Acad. Sci.. 109, (26 ), E1679 –E1687 (2012).CrossRef
Golbabaee  M., and Vandergheynst  P., “Hyperspectral image compressed sensing via low-rank and joint-sparse matrix recovery,” in  IEEE Int. Conf. on Acoustics, Speech and Signal Processing , pp. 2741 –2744 (2012).
Saint-Cyr  M.  et al., “In-vivo quantitative evaluation of perfusion zones and perfusion gradient in the deep inferior epigastric artery perforator flap,” Proc. SPIE. 8618, , 861806  (2013). 0277-786X CrossRef
USDA-FSIS, “Detection and isolation of non-O157 Shiga-toxin producing Escherichia coli (STEC) from meat products and carcass and environmental sponges,” Laboratory Guidebook: MLG 5B.05, Athens, GA (2014).
Yoon  S. C., , Lawrence  K. C., and Park  B., “Automatic counting and classification of bacterial colonies using hyperspectral imaging,” Food Bioprocess Technol.. (2015).CrossRef

Seung-Chul Yoon is a research electronics engineer at the U.S. Department of Agriculture, Agricultural Research Service (USDA ARS). He received his BS and MS degrees in electronics engineering from Yonsei University, Seoul, Korea, and his PhD in electrical engineering from the University of Illinois at Urbana–Champaign, respectively. His research areas include development of imaging technology to solve food safety and quality problems. His expertise is in hyperspectral imaging, machine vision, and real-time imaging.

Tae-Sung Shin is an IT specialist in the Quality and Safety Assessment Research Unit of USDA ARS. He received his PhD in statistics from Iowa State University and joined StatSoft to develop statistical and data mining modules for their flagship product STATISTICA as a senior scientist and then as manager of analytical solutions. He was also hired as a senior software engineer developing business intelligence systems at the information analytics division of Caterpillar before working at USDA ARS.

Kurt C. Lawrence is the research leader of the Quality and Safety Assessment Research Unit with USDA ARS at the U.S. National Poultry Research Center in Athens, Georgia. He has been with USDA for 29 years and has a PhD (1997) in biological and agricultural engineering from the University of Georgia. His research involves hyperspectral and other imaging methods to enhance food safety and quality in the poultry industry.

Gerald W. Heitschmidt is a physical scientist (spectral imaging) with USDA ARS. He received his BS and MA degrees in geography from Kansas State University with an emphasis on remote sensing. His research areas include hyperspectral, multispectral, and color imaging.

Bosoon Park is a research agricultural engineer at USDA ARS in Athens, Georgia. He received his PhD from Texas A&M University. His research interests include food-related nanotechnology, hyperspectral and real-time multispectral imaging, hyperspectral microscope imaging, and Raman spectroscopy for chemical and biological hazards. He is an associate editor of Transactions of the ASABE and an editorial board member for the Journal of Food Measurement and Characterization.

Gary R. Gamble is a research chemist at USDA ARS. He received his BS degree in chemistry from New Mexico State University, Las Cruces, New Mexico, and his PhD in chemical physics from Washington State University, Pullman, Washington. His primary research focus is on the development of chemical and spectroscopic technologies to address food safety problems. His expertise is in analytical chemistry and spectroscopy.

© The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Citation

Seung-Chul Yoon ; Tae-Sung Shin ; Kurt C. Lawrence ; Gerald W. Heitschmidt ; Bosoon Park, et al.
"Hyperspectral imaging using RGB color for foodborne pathogen detection", J. Electron. Imaging. 24(4), 043008 (Aug 10, 2015). ; http://dx.doi.org/10.1117/1.JEI.24.4.043008


Figures

Graphic Jump Location
Fig. 1
F1 :

Hyperspectral imaging system.

Graphic Jump Location
Fig. 2
F2 :

Hyperspectral reflectance image mosaics (color-composites): (a) training set and (b) test set. The hyperspectral images of each of six serogroups were arranged into one row from top to bottom. The images in each column were the cultures of the same cell dilution, as denoted by the target colony forming unit (CFU).

Graphic Jump Location
Fig. 3
F3 :

Spectral sensitivities of a color CCD sensor (Sony ICX445AQ) on Prosilica GC1290C.

Graphic Jump Location
Fig. 4
F4 :

Plate binary mask mosaics of (a) training set and (b) test set.

Graphic Jump Location
Fig. 5
F5 :

Region-of-interest (ROI) image mosaics of (a) training set and (b) test set.

Graphic Jump Location
Fig. 6
F6 :

ROI examples are (a) plate mask ROI (MASK), (b) 5% random samples (MASK5%), (c) colony ROIs plus random pixels (COLONY), (d) 5-colony pixels plus random pixels (COLONY5p), (e) mean spectra (denoted at the center of each colony in the image) of each colony ROI plus random pixels (COLONYa), and (f) enlarged colony ROIs and random pixels (COLONYd).

Graphic Jump Location
Fig. 7
F7 :

RGB color images simulated by a camera sensitivity function (Sony ICX445AQ CCD): (a) training set and (b) test set.

Graphic Jump Location
Fig. 10
F10 :

Mean reflectance spectra obtained by the regression models for each STEC serogroup [(a) O26, (b) O45, (c) O103, (d) O111, (e) O121, and (f) O145)]: trained with colony ROIs and some agar samples (ROI set COLONY) in the training set and validated on colonies in the test set.

Graphic Jump Location
Fig. 9
F9 :

Mean reflectance spectra obtained by the regression models for each Shiga toxin-producing Escherichia coli (STEC) serogroup [(a) O26, (b) O45, (c) O103, (d) O111, (e) O121, and (f) O145)]: trained with all pixels on plate masks (ROI set MASK) in the training image set and validated on colonies in the test set.

Graphic Jump Location
Fig. 8
F8 :

Prediction performance of regression models. p is a polynomial order. With the test set, (a) the average R2 values were computed for all test samples over all wavelengths, and (b) the average root mean square deviation values were computed for all test samples at each wavelength.

Graphic Jump Location
Fig. 11
F11 :

(a) Color-composite images obtained from reconstructed hyperspectral images and (b) pseudo-color images derived from the normalized sum of differences between the measured and predicted hyperspectral images over all wavelengths.

Tables

Table Grahic Jump Location
Table 1Goodness of fit001 of regression models in relation to spectral sensitivity functions.
Table Footer NoteaAverage values over 400 to 1000 nm. The number in () is the average over 400 to 700 nm.
Table Grahic Jump Location
Table 2Goodness of fits002 of regression models in relation to sample sizes.
Table Footer NoteaAverage values over 400 to 1000 nm. The number in () is the average over 400 to 700 nm.
Table Grahic Jump Location
Table 3Overall classification accuracy of classification models observed at the colony level.

References

Yoon  S. C.  et al., “Hyperspectral reflectance imaging for detecting a foodborne pathogen: Campylobacter,” Trans. ASABE. 52, (2 ), 651 –662 (2009).CrossRef
Yoon  S. C.  et al., “Detection of Campylobacter colonies using hyperspectral imaging,” Sens. Instrum. Food Qual. Saf.. 4, (1 ), 35 –49 (2010).CrossRef
Yoon  S. C.  et al., “Hyperspectral imaging for differentiating colonies of non-O157 Shiga-toxin producing Escherichia coli (STEC) serogroups on spread plates of pure cultures,” J. Near Infrared Spectrosc.. 21, (2 ), 81 –95 (2013).CrossRef
Windham  W. R.  et al., “Hyperspectral imaging of Shiga toxin-producing Escherichia coli serogroups O26, O45, O103, O111, O121, and O145 on Rainbow agar,” J. Food Prot.. 76, (7 ), 1129 –1136 (2013). 0362-028X CrossRef
Yoon  S. C.  et al., “Differentiation of big-six non-O157 Shiga-toxin producing Escherichia coli (STEC) on spread plates of mixed cultures using hyperspectral imaging,” Food Meas.. 7, (2 ), 47 –59 (2013).CrossRef
Westland  S., , Ripamonti  C., and Cheung  V., Computational Colour Science Using MATLAB. , 2nd ed.,  Wiley ,  Hoboken, New Jersey  (2012).
López-Alvarez  M. A.  et al., “Selecting algorithms, sensors, and linear bases for optimum spectral recovery of skylight,” J. Opt. Soc. Am. A. 24, (4 ), 942 –956 (2007).CrossRef
Tsumura  N., , Miyake  Y., and Bochko  V., “Spectral color imaging system for estimating spectral reflectance of paint,” J. Imaging Sci. Technol.. 51, (1 ), 70 –78 (2007). 1062-3701 CrossRef
Haneishi  H.  et al., “System design for accurately estimating the spectral reflectance of art paintings,” Appl. Opt.. 39, (35 ), 6621 –6632 (2000). 0003-6935 CrossRef
Stigell  P., , Miyata  K., and Hauta-Kasari  M., “Wiener estimation method in estimating of spectral reflectance from RGB images,” Pattern Recognit. Image Anal.. 17, (2 ), 233 –242 (2007). 1054-6618 CrossRef
Imai  F. H., and Berns  R. S., “Spectral estimation using trichromatic digital cameras,” in  Proc. of the Int. Symp. on Multispectral Imaging and Color Reproduction for Digital Archives ,  Chiba, Japan , pp. 42 –49 (1999).
Valero  E. M.  et al., “Recovering spectral data from natural scenes with an RGB digital camera and colored filters,” Color Res. Appl.. 32, (5 ), 352 –360 (2007).CrossRef
Kawakami  R.  et al., “High-resolution hyperspectral imaging via matrix factorization,” in  IEEE Conf. on Computer Vision and Pattern Recognition , pp. 2329 –2336 (2011).
Guo  Z., , Wittman  T., and Osher  S., “L1 unmixing and its application to hyperspectral image enhancement,” Proc. SPIE. 7334, , 73341M  (2009). 0277-786X CrossRef
Studer  V.  et al., “Compressive fluorescence microscopy for biological and hyperspectral imaging,” Proc. Natl. Acad. Sci.. 109, (26 ), E1679 –E1687 (2012).CrossRef
Golbabaee  M., and Vandergheynst  P., “Hyperspectral image compressed sensing via low-rank and joint-sparse matrix recovery,” in  IEEE Int. Conf. on Acoustics, Speech and Signal Processing , pp. 2741 –2744 (2012).
Saint-Cyr  M.  et al., “In-vivo quantitative evaluation of perfusion zones and perfusion gradient in the deep inferior epigastric artery perforator flap,” Proc. SPIE. 8618, , 861806  (2013). 0277-786X CrossRef
USDA-FSIS, “Detection and isolation of non-O157 Shiga-toxin producing Escherichia coli (STEC) from meat products and carcass and environmental sponges,” Laboratory Guidebook: MLG 5B.05, Athens, GA (2014).
Yoon  S. C., , Lawrence  K. C., and Park  B., “Automatic counting and classification of bacterial colonies using hyperspectral imaging,” Food Bioprocess Technol.. (2015).CrossRef

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Related Book Chapters

Topic Collections

Advertisement
  • Don't have an account?
  • Subscribe to the SPIE Digital Library
  • Create a FREE account to sign up for Digital Library content alerts and gain access to institutional subscriptions remotely.
Access This Article
Sign in or Create a personal account to Buy this article ($20 for members, $25 for non-members).
Access This Proceeding
Sign in or Create a personal account to Buy this article ($15 for members, $18 for non-members).
Access This Chapter

Access to SPIE eBooks is limited to subscribing institutions and is not available as part of a personal subscription. Print or electronic versions of individual SPIE books may be purchased via SPIE.org.