JEI Letters

Blemish detection in camera production testing using fast difference filtering

[+] Author Affiliations
Leena Lepistö, Jarno Nikkanen, Matti Suksi

Nokia Corporation, Visiokatu 3, FI-33720 Tampere, Finland

J. Electron. Imaging. 18(2), 020501 (May 11, 2009). doi:10.1117/1.3132004
History: Received December 01, 2008; Revised March 24, 2009; Accepted March 27, 2009; Published May 11, 2009
Text Size: A A A

Open Access Open Access

In the camera manufacturing, special methods are needed to detect blemishes occurring on the camera sensor pixels. A blemish is referred as a region of pixels in the camera sensor that are somewhat darker than the background. The blemishes are difficult to detect accurately, but on the other hand, they cause a significant reduction in camera quality. We present a novel filtering method for the blemish detection. The method is based on image scaling, filtering, and difference image calculation that is very fast and accurate in the detection of blemishes. In addition, the algorithm can cope with unprocessed raw image data, in which various distortions, such as noise and vignetting, can be present.

Figures in this Article

Automated visual inspection systems are widely used in almost all fields of industry for inspecting goods and materials, including, e.g., paper, metal, wood, and textiles.1 The use of various filtering techniques is common in automated visual inspection tasks, for example, by using various texture filters.1 A number of examples can be found on this, for example, in rock surface characterization.2 One growing area of the blemish detection is the manufacturing process of flat liquid-crqstal device displays3 and light-emitting device chips.4

The application area of this paper is the production testing of digital camera sensors. The number of pixels in high-resolution complementary metal oxide semiconductor camera sensors has increased rapidly from less than one million to eight million and over the past few years. In the camera manufacturing, it is necessary to inspect the quality of the camera by inspecting anomalies on the camera sensors that potentially are able to reduce the image quality given by the camera. Because of the increased pixel numbers, the quality inspection of the sensors has also become more time consuming. In addition, the huge production volumes have caused a need for extremely fast online inspection. The inspection systems must be able to accurately detect the anomalies, but not to find false defects in order to maintain as good a yield as possible.

In camera production testing, two most common defect types are defected pixels and blemishes. The defected pixels include hot pixels and dead pixels, which are typically significantly brighter or darker pixels than their neighborhood. In practice, dead pixels are more interesting in the current testing. A blemish is defined as a cluster of very mild dead pixels or a dark area caused by, e.g., dust or other objects inside a camera module. Both of these anomalies have a certain impact on the image quality given by the camera. The impact depends usually on the size and location of the defect on the sensor region; single dead pixels at the corner regions are not always critical, but the blemishes are typically more serious defects whose occurrence yields to the rejection of the camera.

The detection of the defects in the camera sensor is based on an image taken of a bright flat surface by the camera being tested. Because the dead pixel is like an impulse, it is relatively straightforward to detect by using an impulse-sensitive filtering approach. The blemishes, on the other hand, are much more difficult to detect because their edges are smooth and slow changes in the image signal, instead of impulselike dead pixels. Because of this difference, it is very difficult to detect these two defect types by using the same filter. Instead, two different methods are typically needed, which, in turn, increases the testing time. The images used in the production testing are typically raw sensor data that are not processed in any way, and consequently, several kinds of artifacts can be present in the data. Typically, the noise and vignetting artifact are the most serious challenges for the detection algorithms. The vignetting means a phenomenon of radial falloff of the image intensity from the image center.5 Hence, the edges and corners of the image become darker than the image central area. This kind of slowly changing average is able to cause problems to many blemish detection algorithms because the edges of the blemish regions are very similar as in the case of vignetting. Figure 1 shows an example of a typical production test image of resolution 2074×1558pixels. In this image, vignetting artifact is clearly visible.

Graphic Jump LocationF1 :

An example of a production test image with several blemishes and dead pixels (zoomed). There is also a clearly visible vignetting artifact in the image.

A straightforward method for the blemish detection would be to use, e.g., a differential filter that tries to detect the slowly ascending and descending signal values that could be interpreted as blemish edges. However, preliminary testing with real production test data revealed that these kinds of approaches are ineffective. This is because all these methods are sensitive to vignetting, which makes the blemish detection inaccurate. Also, the noise occurring in the image often causes difficulties in the blemish detection. In addition, separate filters must be used for the detection of dead pixels and blemishes, due to the different natures of these defects described in Section 1.

To overcome these drawbacks, we present a new, alternative solution for the blemish detection. The method has four major steps: (i) image size reduction by scaling, (ii) blemish detection by filtering, (iii) image subtraction, and (iv) thresholding. The method is based on the fact that the blemish regions are typically quite large compared to the actual image size. Consequently, by selecting an appropriate image-scaling method, image size can be significantly reduced, which also reduces the effect of noise. A scaling method used in this paper can be seen in Fig. 2, where the sample pixels are selected from the image with 16pixel distance from each other. In this manner, a smaller image is formed by using an average value of every other pixel in this neighborhood to represent the pixel value in the scaled image. The scaling can be made in the same loop as the dead pixel detection and just save the scaling values into memory. When the scaling is made for every 16th pixel in both vertical and horizontal direction, the scaled image has only 0.4% of the pixel number of the original image. The size of the neighborhood was decided based on preliminary experiments, in which the effects of different neighborhood sizes on the noise levels were compared.

Graphic Jump LocationF2 :

The idea of scaling: the average value of every other pixel in 16×16pixel neighborhood is used to represent selected pixels in a new, smaller image.

As soon as the whole image is scaled, it can be used for blemish detection. A good approach for this would be to use an efficient mean or median filter to reduce the effect of the blemishes and then calculate a difference image between the filtered and original frame. In this paper, however, we present a more efficient way of making this by using a novel type of filtering approach that also calculates the difference image on the fly. The filtering approach aims at detecting the drops of intensity profile caused by a blemish as illustrated in Fig. 3. The filter presented in Fig. 3 is applied on the rows y of the frame I(x,y), and the filtering result is calculated by means of two median values calculated at distance w from each other. The distance w is selected to be large enough to cover the width of potential blemishes, but it should not be too long to avoid the possible disturbances caused by vignetting. The suggested value for w is experimentally selected by inspecting the effect of value w on the difference signal,Display Formula

Mx,yL=median[I(xm1,y1),I(xm1,y),I(xm1,y+1)],
Display Formula
xm1=xw2,
Display Formula
Mx,yR=median[I(xm2,y1),I(xm2,y),I(xm2,y+1)],
Display Formula
xm2=x+w2.
Once the median values are calculated, the output of the filtering is considered as an average of these twoDisplay Formula
2Î(x,y)=12(Mx,yL+Mx,yR)
Then the difference is calculated. Only the positive values of the difference between the filtered image and the original one are considered (that is, the blemish region always has a lower intensity than its surroundings),Display Formula
3Idifference(x,y)=max[0,Î(x,y)I(x,y)].
The filtering method itself is very fast because it only averages two median values calculated from three values each. If w is correctly selected, then this produces very accurate results and typically does not produce false detections. It is obvious that the filtering method is not sensitive to vignetting, since there will be no significant value for Idifference in areas where the intensity only decreases or increases, as shown in Fig. 3. Because the effect of noise is already attenuated by scaling method, it is possible to use very small number of pixels in the calculation of the values ML and MR.

Graphic Jump LocationF3 :

(a) Intensity profile for line y and (b) the principle of the filter.

The blemish regions can be extracted from the obtained difference image Idifference by thresholding. One way of doing the thresholding is to use an adaptive threshold value that is dependent on the difference image, when it is more robust to differences in, e.g., exposure and sensitivity. This can be made by using a threshold proportional to the standard deviation of the whole difference image. The threshold value can be then written as T=As, where constant A is experimentally tuned. It is also possible to use a fixed threshold by using two fixed threshold values, upper and lower thresholds. If a pixel is over upper threshold, then its eight neighbor pixels are averaged, and if this average value is over lower threshold, then the pixel is marked as a blemish. By using this two-threshold technique, weaker blemishes can also be detected more effectively. The tuning of the thresholds can be based on, e.g., the intensity of the central area of the image, based on which it is easy to decide the constant threshold experimentally.

In the experiments, a test set containing 25 images taken with different sensors and varying exposure times, was used. Part of the images were significantly under- or overexposed. In most of these images, there were several blemishes. Once the algorithm was properly tuned, it could easily detect all the blemishes in the test set. In the tuning, selected sample images from a larger collection were used. Basically, the results show that once the algorithm parameters, namely, the threshold value, are properly tuned the algorithm is able to effectively detect the blemishes occurring in a certain sensor type. Also the blemish region edges could be localized accurately. An example of the blemish detection procedure can be seen in Fig. 4.

Graphic Jump LocationF4 :

Filtering procedure for the example image presented in Fig. 1. From left to right: (a) Scaled image of resolution 64×48pixels, (b) filtering result obtained from the proposed filtering method, and (c) thresholded image.

In this paper, a novel method for blemish detection was presented. In this method, the image size is first significantly reduced with an appropriate scaling method that also effectively attenuates noise. After that, the blemish regions are extracted by the filtering method developed in this paper. This method makes the filtering and includes also the subtraction from the original frame. Hence, the memory requirement is minimized and separate calculation for difference image is not needed. This way, the blemish regions will have the highest values in the difference signal, and finally, it is possible to extract these regions from the difference image by thresholding.

Compared to the most other filtering-based methods, the presented approach has several benefits. First, it is not sensitive to vignetting or noise present in the image. Second, it is easy to implement the blemish-detection algorithm as a part of dead pixel detection. In practice, the proposed filter can detect both dead pixels and blemishes, because the dead pixels can be found by setting the filter width to three in the original image. If brighter values are also allowed [Eq. 3 can go negative], then it works also for hot pixels. Thus, the scaling operation necessary to the blemish detection can be a part of the dead pixel detection loop that is made pixel by pixel. Third, thanks to the very small image size used and the developed filtering method, the computational time required by the algorithm is very low. Especially, the computational time for blemish detection can be made very low, the current implementation of blemish detection (including scaling) requires only 7% of the time required for the dead pixel detection by using the same filter. In general, the proposed method makes the overall testing time of both dead pixels and blemishes greatly shorter on the production line than that of previously used method that detects merely the dead pixels.

Pietikäinen  M., , Ojala  T., , and Silven  O., “ Approaches to texture-based classification, segmentation and surface inspection. ,” in Chen  C. H., Pau  L. F., and Wang  P. S. P., Eds.,  Handbook of Pattern Recognition and Computer Vision. , 2nd ed., pp. 711–736 ,  World Scientific , Singapore ((1999)).
Lepistö  L., , Kunttu  I., , and Visa  A., “ Rock image classification using color features in Gabor space. ,” J. Electron. Imaging.  1017-9909 , 14, (4 ), 040503  ((2005)).
Tsai  D.-M., and Chuang  S.-T., “ 1D-based defect detection in patterned TFT-LCD panels using characteristic fractal dimension and correlations. ,” Mach. Vision Appl..  0932-8092  (in press).
Lin  H.-D., , Chung  C.-Y., , and Chiu  S. W., “ Computer-aided vision system for surface blemish detection of LED chips. ,” in  Proc. Int. Conf. Adaptive and Natural Computing Algorithms (ICANNGA 2007), Part II. , Warsaw, Poland, 11–14 April 2007, Beliczynski  B., Dzielinski  A., Iwanowski  M., and Ribero  B. (Eds.), Lecture Notes in Computer Science Vol. 4432, pp. 525–533 ,  Springer , Berlin ((2007)).
Reza  M. T., “ Vignetting artifact reduction in an efficient way for digital camera image. ,” MSc thesis, Tampere University of Technology ((2008)).
© 2009 SPIE and IS&T

Citation

Leena Lepistö ; Jarno Nikkanen and Matti Suksi
"Blemish detection in camera production testing using fast difference filtering", J. Electron. Imaging. 18(2), 020501 (May 11, 2009). ; http://dx.doi.org/10.1117/1.3132004


Figures

Graphic Jump LocationF1 :

An example of a production test image with several blemishes and dead pixels (zoomed). There is also a clearly visible vignetting artifact in the image.

Graphic Jump LocationF2 :

The idea of scaling: the average value of every other pixel in 16×16pixel neighborhood is used to represent selected pixels in a new, smaller image.

Graphic Jump LocationF3 :

(a) Intensity profile for line y and (b) the principle of the filter.

Graphic Jump LocationF4 :

Filtering procedure for the example image presented in Fig. 1. From left to right: (a) Scaled image of resolution 64×48pixels, (b) filtering result obtained from the proposed filtering method, and (c) thresholded image.

Tables

References

Pietikäinen  M., , Ojala  T., , and Silven  O., “ Approaches to texture-based classification, segmentation and surface inspection. ,” in Chen  C. H., Pau  L. F., and Wang  P. S. P., Eds.,  Handbook of Pattern Recognition and Computer Vision. , 2nd ed., pp. 711–736 ,  World Scientific , Singapore ((1999)).
Lepistö  L., , Kunttu  I., , and Visa  A., “ Rock image classification using color features in Gabor space. ,” J. Electron. Imaging.  1017-9909 , 14, (4 ), 040503  ((2005)).
Tsai  D.-M., and Chuang  S.-T., “ 1D-based defect detection in patterned TFT-LCD panels using characteristic fractal dimension and correlations. ,” Mach. Vision Appl..  0932-8092  (in press).
Lin  H.-D., , Chung  C.-Y., , and Chiu  S. W., “ Computer-aided vision system for surface blemish detection of LED chips. ,” in  Proc. Int. Conf. Adaptive and Natural Computing Algorithms (ICANNGA 2007), Part II. , Warsaw, Poland, 11–14 April 2007, Beliczynski  B., Dzielinski  A., Iwanowski  M., and Ribero  B. (Eds.), Lecture Notes in Computer Science Vol. 4432, pp. 525–533 ,  Springer , Berlin ((2007)).
Reza  M. T., “ Vignetting artifact reduction in an efficient way for digital camera image. ,” MSc thesis, Tampere University of Technology ((2008)).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Related Book Chapters

Topic Collections

Advertisement
  • Don't have an account?
  • Subscribe to the SPIE Digital Library
  • Create a FREE account to sign up for Digital Library content alerts and gain access to institutional subscriptions remotely.
Access This Article
Sign in or Create a personal account to Buy this article ($20 for members, $25 for non-members).
Access This Proceeding
Sign in or Create a personal account to Buy this article ($15 for members, $18 for non-members).
Access This Chapter

Access to SPIE eBooks is limited to subscribing institutions and is not available as part of a personal subscription. Print or electronic versions of individual SPIE books may be purchased via SPIE.org.