The surface machining of cracks is a key issue to ensure the quality of steel rods and billets. The aim is to grind these defects out of the material. This paper presents a real-time optical servo-system, consisting of three image processing systems and an industrial robot, which fully automate this process. A high resolution color progressive scan camera, placed at a suitable position above the roller conveyor, observes the material and detects color markings indicating the presence of a crack. This camera system controls the roller conveyor transporting the material until a marked crack is detected. Diffuse light sources provide homogeneous lighting to ensure reliable detection of the markings. A demosaicing algorithm, RGB to HSL color modeling and thresholding with statistical morphology are used to identify the marked areas. On detecting a crack the material is automatically positioned within the working area of an industrial robot. A collineation is used to generate metric two-dimensional coordinates corresponding to the bounding rectangle of the detected error. At this point two plane-of-light scanners are used to acquire a cross section of the material to the left and the right of the robot's working area. From this, a three-dimensional model for the rod or billet surface is calculated and the two-dimensional coordinates of the color marking are projected onto this surface to generate a patch. The coordinates of this patch are sent to the 6R industrial robot, which then grinds out the defect. A new concept has been implemented which enables the calibration of the three image processing systems and the industrial robot so as to have one common coordinate system. Operational results have shown the full functionality of the system concept in the harsh environment of a steel production facility.

We studied the statistical learning methods with imbalanced training data sets. Imbalanced training sets are very common in industrial machine vision applications. The minority class contains the defects or anomalies we try to catch. The majority class contains the "regular" objects. We need a method that performs well at both false positive and false negative error rates. Traditional methods such as classification tree yield unsatisfactory results. We propose a two-stage classification scheme. We first use a subset selection method to remove redundant examples from the majority class. As a result, the training sample becomes more balanced without losing critical boundary information. The computation-intensive methods such as boosted classification trees are then applied to further improve both error rates.

A challenge in the semiconductor industry is the 3D inspection of solder bumps grown on wafers for direct die-to-die bonding. In an earlier work we proposed a novel mechanism for reconstructing wafer bump surface in 3D, which is based upon projecting a binary pattern to the surface and capturing image of the illuminated scene. By shifting the binary pattern in space and every time taking a separate image of the illuminated surface, each position on the illuminated surface will be attached with a binary code in the sequence of images taken. 3D information about the bump surface can then be obtained over these coded points via triangulation. However, when a binary pattern is projected onto the inspected surface through projection lenses, the high order harmonics of the pattern are often diminished because of the lens' limited bandwidth. This will lead to blurring of the projected fringe boundaries in the captured image data and make differentiation between dark and bright fringes there difficult. In addition, different compositions of the target surface, some metallic (the solder surface) and some not (the substrate surface of the wafer), have different reflectance functions (including both the specular and lambertian components). This makes fringe boundary detection in the image data an even more challenging problem. This paper proposes a solution to the problem. It makes use of the spatial-temporal image volume over the target surface to tackle the issue of inhomogeneous reflectance function. It is shown that the observed intensity profile across the images of a fixed point has the same up-and-down profile of the orignal binary gratings, regardless of the reflectance on the target surface, from which edges can be detected using classical methods like the gradient based ones. Preliminary study through theoretical analysis and empirical experiments on real image data demonstrate the feasibility of proposed approach.

The shrunk dimension of electronic devices leads to more stringent requirement on process control and quality assurance of their fabrication. For instance, direct die-to-die bonding requires placement of solder bumps not on PCB but on the wafer itself. Such wafer solder bumps, which are much miniaturized from the counterparts on PCB, still need to have their heights meet the specification, or else the electrical connection could be compromised, or the dies be crushed, or even the manufacturing equipments be damaged. Yet the tiny size, typically tens of microns in diameter, and the textureless and mirror nature of the bumps pose great challenge to the 3D inspection process. This paper addresses how a large number of such wafer bumps could have their heights massively checked against the specification. We assume ball bumps in this work. We propose a novel inspection measure about the collection of bump heights that possesses these advantages: (1) it is sensitive to global and local disturbances to the bump heights, thus serving the bump height inspection purpose; (2) it is invariant to how individual bumps are locally displaced against one another on the substrate surface, thus enduring 2D displacement error in soldering the bumps onto the wafer substrate; and (3) it is largely invariant to how the wafer itself is globally positioned relative to the imaging system, thus having tolerance to repeatability error in wafer placement. This measure makes use of the mirror nature of the bumps, which used to cause difficulty in traditional inspection methods, to capture images of two planes. One contains the bump peaks and the other corresponds to the substrate. With the homography matrices of these two planes and fundamental matrix of the camera, we synthesize a matrix called Biplanar Disparity Matrix. This matrix can summarize the bumps' heights in a fast and direct way without going through explicit 3D reconstruction. We also present a design of the imaging and illumination setup that allows the measure to be revealed in two images, and how the inspection measure could be estimated from the image data so acquired. Both synthetic and real data experimental results are shown to illustrate the effectiveness of the proposed system.

A machine-vision system for real-time classification of moving objects with highly reflective metallic surfaces and complex 3D-structures is presented. As an application example of our Three-Color Selective Stereo Gradient Method (Three-Color SSGM) a classification system for the three main coin denominations of Euro coins is presented. The coins are quickly moving in a coin validation system. The objective is to decide only from comparison of measured 3D-surface properties with characteristic topographical data stored in a database whether a coin belongs to one of the reference classes or not. Under illumination of a three-color LED-ring a single image of the moving coin is captured by a CCD-camera. Exploiting the spectral properties of the illumination sources, which correspond to the special spectral characteristics of the camera, three independent subimages can be extracted from the first. Comparison between these subimages leads to a discrimination between a coin with real 3D-surface and a photographic image of a coin of the same type. After the coin has been located and segmented, grey value based rotation and translation invariant features are extracted from a normalized image. In combination with template matching methods, a coin can be classified. Statistical classification results will be reported.

This article proposes to deal with noisy and variable size color textures. It also proposes to deal with quantization methods and to see how such methods change final results. The method we use to analyze the robustness of the textures consists of an auto-classification of modified textures. Texture parameters are computed for a set of original texture samples and stored into a database. Such a database is created for each quantization method. Textures from the set of original samples are then modified, eventually quantized and classified according to classes determined from a precomputed database. A classification is considered incorrect if the original texture is not retrieved. This method is tested with 3 textures parameters: auto-correlation matrix, co-occurrence matrix and directional local extrema as well as 3 quantization methods: principal component analysis, color cube slicing and RGB binary space slicing. These two last methods compute only 3 RGB bands but could be extended to more. Our results show that, with or without quantization, autocorrelation matrix parameter is less sensitive to noise and to scaling than the two other tested texture parameters. This implies that autocorrelation matrix should probably be preferred for texture analysis with non controlled condition, typically industrial applications where images could be noisy. Our results also shows that PCA quantization does not change results where the two other quantization methods change them dramatically.

The ARS imaging research group in Athens, Georgia has developed a real-time multispectral imaging system for fecal and ingesta contaminant detection on broiler carcasses for poultry industry. The industrial scale system includes a common aperture camera with three visible wavelength optical trim filters. This paper demonstrates calibration of common aperture multispectral imaging hardware and real-time image processing software. The software design, especially the Unified Modeling Language (UML) design approach was used to develop real-time image processing software for on-line application. The UML models including class, object, activity, sequence, and collaboration diagram were presented. Both hardware and software for a real-time fecal and ingesta contaminant detection were tested at the pilot-scale poultry processing line. The test results of industrial sacle real-time system showed that the multispectral imaging technique performed well for detecting fecal contaminants with a commercial processing speed (currently 140 birds per minute). The accuracy for the detection of fecal and ingesta contaminates was approximately 96%.

For the homogenisation of the molten steel it is necessary to rinse the melting bath. Therefore two porous plugs are installed in the bottom of the casting ladle through which the gas is blown into the ladle. The movement of the melting surface is chaotic. Other process stages, which are distortions to the image processing system, like steam or mechanical parts moving within the scene have to be taken into consideration too. Standard straight forward analytic algorithms fail. The uncertainties cannot be handled in a proper way. We decided to use a RGB binary converter followed by a fuzzy classifier. If the flushing is active molten steel breaks through the slag. This molten steel areas show a certain colour spectrum. The RGB-binary conversation is necessary to detect the molten cast breaking the slag. The size of these colour areas is direct proportional to the intensity of the flushing. The fuzzy block felts the results of the binary conversation and splits them into the intensity grades. This method allows the detection of five stages of the flushing under the given conditions at the melting process and it is able to detect steam or other disturbing parts moving through the scene as well.

The characterization of commercial 3D scanners allows acquiring precise and useful data. The accuracy of range and, more recently, color for 3D scanners is usually studied separately, but when the 3D scanner is based on structured light with a color coding pattern, color influence on range accuracy should be investigated. The commercial product that we have tested has the particularity that it can acquire data under ambient light instead of a controlled environment as it is with most available scanners. Therefore, based on related work in the literature and on experiments we have done on a variety of standard illuminants, we have designed an interesting setup to control illuminant interference. Basically, the setup consists of acquiring the well-known Macbeth ColorChecker under a controlled environment and also ambient daylight. The results have shown variations with respect to the color. We have performed several statistical studies to show how the range results evolve with respect to the RGB and the HSV channels. In addition, a systematic noise error has also been identified. This noise depends on the object color. A subset of colors shows strong noise errors while other colors have minimal or even no systematic error under the same illuminant.

Industrial reproduction, as stereography or lithography, have a lack in texture information, as they only deal with 3D reconstruction. In this paper, we provide a new technique to map texture on real 3D objects, by synthesizing a novel view from two camera images to a projector frame, considered as a camera acting in reverse. No prior information on the pose or the shape of the 3D object is necessary, however hard calibration of the complete system is needed.

A model and a method providing a 3D reconstruction of a given translucent object from a series of image acquisitions performed with various focus tunings is proposed. The object is imaged by transmission; refraction, reflection and diffusion effects are neglected. It is modeled as a stack of translucent parallel planes whose positions are denoted by their abscissa. A first degree order discrete expansion of the transmission coefficient is proposed. The optical device is assumed to be linear, its blurring function depends on the focusing distance. All the images acquired with various focusing distances are lexicographically sorted in a vector S and, calling E the images of the absorption coefficients of the object, the acquisition process can be described by a set of linear equations: S = HE. Taking advantage from the peculiar structure of H we propose a very efficient inversion technique with a complexity O(n) allowing practical applications with simple lap top computer in very reasonable time. Examples of results with a simulated 3D translucent object are presented. When no perturbation is assumed, the model is reconstructed with no visible error from the blurred acquisitions.

High quality wood sanding machines require information about the wood surface shape in order to control actuators pushing the sanding paper onto the wood surface. In order to improve the quality of the sanding process a 3D measurement system is currently under development within an EU FP6 project. Several constraints (especially panel size and available measurement system volume) prohibit using off-the-shelf measurement systems. The measurement system is based on laser line triangulation and consists of two inclined 675 nm lasers equipped with line diffraction optics, a high resolution camera, and a standard PC all mounted into a rigid frame. Two mirrors direct the laser light onto the wood panel moving beneath the measurement system. The surface resolution is 1.1 mm/pixel and the depth resolution is 0.4 mm/pixel. Measurements are performed at a rate of 25 frames/sec. The projected lines are detected with subpixel accuracy and converted into world (x, y, z) coordinates using calibration data. The measurement accuracy is approximately identical over the full width of the measurement system. In x and z direction the surface resolution is nearly constant. In y direction the resolution depends on the panel shape and speed. In case of shadowing (i. e. when only one line is visible) the resolution is 10 mm otherwise it is 5 mm.

Two measurement systems based on electronic imagers for attitude determination of light and low-cost airborne vectors in clear weather condition are presented. Both the devices are based on the following scheme: a binarized sky line image is selected as referential, then each shot, after binarization is segmented in two parts whose mean values are subtracted to the corresponding data computed from referential image. From this comparison analogical or digital values are deduced that give way to estimating the roll and pitch variation angles. The first system is made of off-the-shelf devices: black and white CMOS camera and processing unit made of a dedicated electronic. A second proposition is presented. It makes use of a CMOS retina, developed by our team, dedicated to real time, analogical intercorrelation computing. A simple external processing implemented in a microcontroller completes the system. A model for the embarked imaging system and an equation for an ideal horizon image with respect to rolling and pitch angles are proposed. The method for estimating these angles from the image and errors induced by the various approximations of the model are then presented. A series of experimental results obtained from real images confirms the previous propositions. The last section is dedicated to a presentation of the retinal imager solution, experimental results are also provided in this part.

Image binarization under non-uniform lighting conditions is required in many industrial machine vision applications. Many local adaptive thresholding algorithms have been proposed in the literature for this purpose. However, existing local adaptive thresholding algorithms are either not robust enough or too expensive for real-time implementation due to very high computation costs. This paper presents a new algorithm for local adaptive thresholding based on a multi-stage framework. In the first stage, a mean filtering algorithm, with kernel-size independent computation cost, is proposed for background modeling to eliminate the non-uniform lighting effect. In the second stage, a background-corrected image is generated based on the background color. In the final stage, a global thresholding algorithm is applied to the background-corrected image. The kernel-size independent computation algorithm reduces the order of computation cost of background modeling from NML² to ML+NL+6NM for an N x M image with an L x L kernel, which enables the real-time processing of objects of arbitrary size. Experiments show that the proposed algorithm performs better than other local thresholding algorithms, such as the Niblack algorithm, in terms of both speed and segmentation results for many machine vision applications under non-uniform lighting conditions.

This paper reports on an image processing algorithm for simultaneous photometric correction and defect detection in semiconductor manufacturing. We note that this problem has some resemblance to change detection in real time image analysis. In particular, the changes between the two images are analogous to the defects in our machine vision system. We therefore applied several detection methods and examined their applicability to defect detection. We first performed a sub-pixel image registration, using a phase correlation method together with a singular value decomposition factorization of the correlation matrix to compute the necessary alignment. We then tested a few change detection methods, including the shading model, derivative model, statistical change detection, linear dependence change detector and Wronskian change detection model. We subjected this system to our collection of raw data acquired from an industrial system, and we evaluated the different methods with respect to the detection accuracy, robustness, and speed of the system. We have promising results at this stage, especially in detecting the blob and line defects that are most commonly found, and when the lighting variation is within a certain threshold.

In this paper, we propose an automatic inspection system, which can automatically detect four types of muras on an LCD panel: cluster mura, v-band mura, rubbing mura, and light leakage mura. To detect cluster muras, the Laplacian of Gaussian (LOG) filter is used. A multi-resolution approach is proposed to detect cluster muras of different scales. To speed up the processing speed, this multi-resolution approach is actually implemented in the frequency domain. To detect v-band muras, we check the variation tendency of the projected 1-D intensity profile. Then, v-band muras are detected by identifying these portions of the 1-D profile where a large deviation occurs. To detect rubbing muras, we designed a frequency mask to detect distinct components in the frequency domain. To detect light leak muras, we apply image mirroring over the boundary parts and adopt the same LOG filter that has been used in detecting cluster muras. All four types of mura detection are integrated together in an efficient way and simulation results demonstrate that this system is indeed very helpful in detecting mura defects.

We describe in this paper new developments in the characterization of coated particle nuclear fuel using optical microscopy and digital imaging. As in our previous work, we acquire optical imagery of the fuel pellets in two distinct manners that we refer to as shadow imaging and cross-sectional imaging. In shadow imaging, particles are collected in a single layer on an optically transparent dish and imaged using collimated back-lighting to measure outer surface characteristics only. In cross-sectional imaging, particles are mounted in acrylic epoxy and polished to near-center to reveal the inner coating layers for measurement. For shadow imaging, we describe a curvaturebased metric that is computed from the particle boundary points in the FFT domain using a low-frequency parametric representation. We also describe how missing boundary points are approximated using band-limited interpolation so that the FFT can be applied. For cross-section imaging, we describe a new Bayesian-motivated segmentation scheme as well as a new technique to correct layer measurements for the fact that we cannot observe the true mid-plane of the approximately spherical particles.

It is difficult to obtain stable roadway images when the camera shakes. In order to solve the problem, we present a two-point calibration method. Two symbol points in sequential frames are selected firstly, if the two points can be matched well in a sequence images, the sequential frames will be matched well. Then the frame difference method is used to segment moving vehicles. Since the moving vehicles signals are very weak, the signals are magnified by a dilation operation. Since the headmost moving vehicle is close to the backmost stationary vehicle on a roadway when traffic light is red, location of the backmost stationary vehicle can be determined by location of headmost moving vehicle. And the backmost stationary vehicle's computer image coordinate can be transformed to world coordinate by a camera model. As a result, the length of stationary vehicle queue can be calculated and estimated. And the ratio between stationary vehicle queue length and roadway length can be obtained. We can use the ratio to evaluate the congestion degree of the roadway.

The detection of varying 2D shapes is a recurrent task for Computer Vision applications, and camera based object recognition has become a standard procedure. Due to the discrete nature of digital images and aliasing effects, shape recognition can be complicated. There are many existing algorithms that discuss the identification of circles and ellipses, but they are very often limited in flexibility or speed or require high quality input data. Our work considers the application of shape recognition for processes in industrial environments and, especially the automatization requires reliable and fast algorithms at the same time. We take a very practical look at the automated shape recognition for common industrial tasks and present a very fast novel approach for the detection of deformed shapes which are in the broadest sense elliptic. Furthermore, we consider the automated recognition of bacteria colonies and coded markers for both 3D object tracking and an automated camera calibration procedure.

This article deals with the problem of the determination of characteristics of imperfect circular objects on discrete images mainly the radius and center's coordinates. Imperfections are provided by discretization, noise and interior distortions present in some production processes. To this end, a multi-level method based on active contours was developed and tested on, noisy or not, misshaped or not, simulated circles whom centers and radius were known. The adequacy of this approach was tested with several methods, among them several Radon based ones. More particularly, this study indicates the relevance of the use of active contours combined with a Radon transform based method, using a description of circles from their tangents, improved thanks to a fitting considering the discrete implementation of Radon transform. Through this study, an active region algorithm based on stationary states of a non linear diffusion principle is proposed. Its originality is to obtain a set of geometric envelopes in one pass, with a correspondence between level threshold of the grayscale result and a regularity scale, more or less close to the original shape. This set of geometric envelopes gives a multiscale representation, from a very regular approximation to a full detailed and roughest representation. Then, a more robust measure of the circle parameters can be computed.

Grain size of forged nickel alloy is an important feature for the mechanical properties of the material. For fully automatic grain size evaluation in images of micrographs it is necessary to detect the boundaries of each grain. This grain boundary detection is influenced directly by artifacts like scratches and twins. Twins can be seen as parallel lines inside one grain, whereas a scratch can be identified as a sequence of collinear line segments that can be spread over the whole image. Both kinds of artifacts introduce artificial boundaries inside grains. To avoid wrong grain size evaluation, it is necessary to remove these artifacts prior to the size evaluation process. For the generation of boundary images various algorithms have been tested. The most stable results were achieved by grayscale reconstruction and a subsequent watershed segmentation. A modified line Hough transform with a third dimension in the Hough accumulator space, describing the distance of the parallel lines, is used to directly detect twins. Scratch detection is done by applying the standard line Hough transform followed by a rule based segment detection along the found Hough lines. The results of these operations give a detection rate of more than 90 percent for twins and more than 50 percent for scratches.

The in vivo imaging of proteins represents a promising technique for understanding the processes taking place at cellular level. Tracking the single proteins manually is tedious and the results are difficult to replicate. Due to the imaging characteristics the automation of the task is difficult. In this paper we study the problem of denoising of the image sequences, spot detection and data association. The 3D version of three denoising algorithms were implemented: adaptive mean filtering, anisotropic diffusion and spatial-tonal convolution. Their effect combined with the spot detection based on the à trous wavelet transform is studied. Finally, a point tracking algorithm is applied having as input the spots detected in the previous step. The algorithm can handle new track creation, track termination as well as one frame occlusions. The paper concludes with a discussion of the results and further work.

This paper presents two spatial methods to discriminate between crop and weeds. The application is related to agronomic image with perspective crop rows. The first method uses a double Hough Transform permitting a detection of crop rows and a classification between crop and weeds. The second method is based on Gabor filtering, a band pass filter. The parameters of this filter are detected from a Fast Fourier Transform of the image. For each method, a weed infestation rate is obtained. The two methods are compared and a discussion concludes about the abilities of these methods to detect the crop rows in agronomic images. Finally, we discuss this method regarding the capability of the spatial approach for classifying weeds from crop.

Detection of circular information including drill holes and inside connection metal rings, plays a key role for the automatic inspection of a multi-layer printed circuit board (PCB). An approach is presented to automatically extract whole circular information from an x-ray image acquired from a multi-layer PCB. By analyzing the x-ray image with a series of some image processing procedures, the basic circular information can be obtained and be treated as an initial contour for further processing. An effective modular active contour is then presented to guide the initial contour to locate the circular information more precisely. Experimental analyses have shown that the proposed approach can reach at the performance of 0.5 pixel average error under 20% random noise added. Experiments on real PCB x-ray images have also confirmed the feasibility of the proposed approach.

We propose a method to detect defects due to spatially non-uniform brightness on LCD panels by using a machine vision technique. The detection method is based on human vision so that proper subjective assessment experiments were conducted to investigate the correlation between the parameters related to non-uniformity and the degree how easily observable it is. The visibility of the defects reveals to depend mainly on the spatial gradient of brightness variation. Thus, in the proposed method, the spatial gradient that is calculated by using extracted contours will be utilized to detect the defects due to non-uniform brightness. The detection method comprises four parts: contour extraction, spatial gradient calculation, decision of defects, and display of defects. We applied the method to the images captured from practical LCD panels with non-uniformity defects and the results were consistent with detection by a human inspector.

Texture characterization and classification remains an important issue in image processing and analysis. Much attention has been focused on methods involving spectral analysis and co-occurrence matrix, as well as more modern approaches such as those involving fractal dimension, entropy and criteria based in multiresolution. The present work addresses the problem of texture characterization in terms of complex networks: image pixels are represented as nodes and similarities between such pixels are mapped as links between the network nodes. It is verified that several types of textures present node degree distributions which are far distinct from those observed for random networks, suggesting complex organization of those textures. Traditional measurements of the network connectivity, including their respective hierarchical extensions, are then applied in order to obtain feature vectors from which the textures can be characterized and classified. The performance of such an approach is compared to co-occurrence methods, suggesting promising complementary perspectives.

Nowadays, visual inspection is very important in the quality control for many industrial applications. However, the complexity of most 3D objects constrains the registration of range images; a complete surface is required to compare the acquired surface to the model. Range finders are very used to digitalize free form shape objects with large resolutions. Moreover, one single view is not enough to reconstruct the whole surface due to occlusions, shadows, etc. In these situations, the motion between reconstructed partial views are required to integrate all surfaces in a single model. However, the use of positioning systems is not always available or adequate due mainly to the size of the objects or the environmental conditions imposed by the precise mechanics which suffer from vibrations present in the industry. In order to solve this problem, a 3D hand sensor is developed to reconstruct 3D objects that let us to compare them with respect the original one.

With the continuous effort of the electronic industry in miniaturizing device size, the task of inspecting the various electrical parts becomes increasingly difficult. For instance, solder bumps grown on wafers for direct die-to-die bonding need to have their 3D shape inspected for assuring electrical contact and preventing damage to the processing equipments or to the dies themselves in the bonding process. Yet, the inspection task is made difficult by the tiny size and the highly specular and textureless nature of the bump surfaces. In an earlier work we proposed a mechanism for reconstructing such highly specular micro-surfaces as wafer bumps. However, the mechanism is capable of recovering 3D positions only. In this paper we describe a new mechanism that recovers surface orientations as well which are as important in describing a surface. The mechanism is based upon projecting light from a point or parallel light source to the inspected surface through a specially designed binary grid. The grid consists of a number of black and transparent blocks, resembling a checker board. By shifting the grid in space a number of times in a direction not parallel to either boundary of the grid elements, and each time taking a separate image of the illuminated surface, we could determine the surface orientations of the inspected surface at points which appear in the image data as grid corners. Experimental results on real objects are shown to illustrate the effectiveness of the proposed mechanism.

The modelling of the shoulder joint is an important step to set a Computer-Aided Surgery System for shoulder prosthesis placement. Our approach mainly concerns the bones structures of the scapulo-humeral joint. Our goal is to develop a tool that allows the surgeon to extract morphological data from medical images in order to interpret the biomechanical behaviour of a prosthesised shoulder for preoperative and peroperative virtual surgery. To provide a light and easy-handling representation of the shoulder, a geometrical model composed of quadrics, planes and other simple forms is proposed.