A vehicle’s license plate is the unique feature by which to identify each individual vehicle. As an important research area of an intelligent transportation system, the recognition of vehicle license plates has been investigated for some decades. An approach based on a visual attention model and deep learning is proposed to handle the problem of Chinese car license plate recognition for traffic videos. We first use a modified visual attention model to locate the license plate, and then the license plate is segmented into seven blocks using a projection method. Two classifiers, which combine the advantages of convolutional neural network-based feature learning and support vector machine for multichannel processing, are designed to recognize Chinese characters, numbers, and alphabet letters, respectively. Experimental results demonstrate that the presented method can achieve high recognition accuracy and works robustly even under the conditions of illumination change and noise contamination.

Constructing appropriate descriptors for interest points in image matching is a critical aspect task in computer vision and pattern recognition. A method as an extension of the scale invariant feature transform (SIFT) descriptor called shape–color alliance robust feature (SCARF) descriptor is presented. To address the problem that SIFT is designed mainly for gray images and lack of global information for feature points, the proposed approach improves the SIFT descriptor by means of a concentric-rings model, as well as integrating the color invariant space and shape context with SIFT to construct the SCARF descriptor. The SCARF method developed is more robust than the conventional SIFT with respect to not only the color and photometrical variations but also the measuring similarity as a global variation between two shapes. A comparative evaluation of different descriptors is carried out showing that the SCARF approach provides better results than the other four state-of-the-art related methods.

We present a software tool called a stereovision egomotion sequence generator that was developed for testing visual odometry (VO) algorithms. Various approaches to single and multicamera VO algorithms are reviewed first, and then a reference VO algorithm that has served to demonstrate the program’s features is described. The program offers simple tools for defining virtual static three-dimensional scenes and arbitrary six degrees of freedom motion paths within such scenes and output sequences of stereovision images, disparity ground-truth maps, and segmented scene images. A simple script language is proposed that simplifies tests of VO algorithms for user-defined scenarios. The program’s capabilities are demonstrated by testing a reference VO technique that employs stereoscopy and feature tracking.

This paper introduces a modified set partitioning in hierarchical trees (SPIHT) algorithm that reduces the number of comparison operations and, consequently, the execution time needed to encode an image as compared to the SPIHT algorithm. The threshold of each independent subband is calculated after applying the discrete wavelet transform to the image. Scanning of the sets inside the subbands is determined by the magnitude of the thresholds that establishes a hierarchical scanning not only for the set of coefficients with larger magnitude, but also for the subbands. The algorithm uses the set partitioning technique to sort the transform coefficients. Results show that the modified SPIHT significantly reduces the number of operations and the execution time without sacrificing visual quality and the PSNR of the recovered image.

It is a challenging task to develop an effective and robust object tracking method due to factors such as severe occlusion, background clutters, abrupt motion, illumination variation, and so on. A tracking algorithm based on weighted subspace reconstruction error is proposed. The discriminative weights are defined based on minimizing reconstruction error with a positive dictionary while maximizing reconstruction error with a negative dictionary. Then a confidence map for candidates is computed through the subspace reconstruction error. Finally, the location of the target object is estimated by maximizing the decision map which combines the discriminative weights and subspace reconstruction error. Furthermore, the new evaluation method based on a forward–backward tracking criterion is used to verify the proposed method and demonstrates its robustness in the updating stage and its effectiveness in the reduction of accumulated errors. Experimental results on 12 challenging video sequences show that the proposed algorithm performs favorably against 12 state-of-the-art methods in terms of accuracy and robustness.

We have devised a full-resolution stereoscopic television system incorporating both a patterned retarder and active glasses. Selective vision of the left image by the left eye and the right image by the right eye is achieved by a conventional combination of a patterned retarder and left and right polarized filters. Full resolution is provided by the active components of the glasses acting as a switchable refractive-type beam displacer. Pairs of line-interleaved images are displayed on an LCD screen sequentially at a frame rate of 120 Hz. With the help of active refraction glasses, the viewer can see full-resolution stereoscopic images as if they are displayed in an interlaced manner. Active glasses are flicker-free. Measured stereoscopic crosstalk is 0.6%, which is defined only by the performance of the patterned retarder.

Conventional image stitching methods were developed under the assumption or condition that (1) the optical center of a camera is fixed (fixed-optical-center case) or (2) the camera captures a plane target (plane-target case). Hence, users should know or test which condition is more appropriate for the given set of images and then select a right algorithm or try multiple stitching algorithms. We propose a unified framework for the image stitching and rectification problem, which can handle both cases in the same framework. To be precise, we model each camera pose with six parameters (three for the rotation and three for the translation) and develop a cost function that reflects the registration errors on a reference plane. The designed cost function is effectively minimized via the Levenberg–Marquardt algorithm. For the given set of images, when it is found that the relative camera motions between the images are large, the proposed method performs rectification of images and then composition using the rectified images; otherwise, the algorithm simply builds a visually pleasing result by selecting a viewpoint. Experimental results on synthetic and real images show that our method successfully performs stitching and metric rectification.

We present Grassmannian sparse representations (GSR), a sparse representation Grassmann learning framework for efficient classification. Sparse representation classification offers a powerful approach for recognition in a variety of contexts. However, a major drawback of sparse representation methods is their computational performance and memory utilization for high-dimensional data. A Grassmann manifold is a space that promotes smooth surfaces where points represent subspaces and the relationship between points is defined by the mapping of an orthogonal matrix. Grassmann manifolds are well suited for computer vision problems because they promote high between-class discrimination and within-class clustering, while offering computational advantages by mapping each subspace onto a single point. The GSR framework combines Grassmannian kernels and sparse representations, including regularized least squares and least angle regression, to improve high accuracy recognition while overcoming the drawbacks of performance and dependencies on high dimensional data distributions. The effectiveness of GSR is demonstrated on computationally intensive multiview action sequences, three-dimensional action sequences, and face recognition datasets.

Binary image descriptors have received a lot of attention in recent years, since they provide numerous advantages, such as low memory footprint and efficient matching strategy. However, they utilize intermediate representations and are generally less discriminative than floating-point descriptors. We propose an image region descriptor, namely local derivative ordinal binary pattern, for object recognition and image categorization. In order to preserve more local contrast and edge information, we quantize the intensity differences between the central pixels and their neighbors of the detected local affine covariant regions in an adaptive way. These differences are then sorted and mapped into binary codes and histogrammed with a weight of the sum of the absolute value of the differences. Furthermore, the gray level of the central pixel is quantized to further improve the discriminative ability. Finally, we combine them to form a joint histogram to represent the features of the image. We observe that our descriptor preserves more local brightness and edge information than traditional binary descriptors. Also, our descriptor is robust to rotation, illumination variations, and other geometric transformations. We conduct extensive experiments on the standard ETHZ and Kentucky datasets for object recognition and PASCAL for image classification. The experimental results show that our descriptor outperforms existing state-of-the-art methods.

Many advanced image-processing softwares are available for tampering images. How to determine the authenticity of an image has become an urgent problem. Copy–move is one of the most common image forgery operations. Many methods have been proposed for copy–move forgery detection (CMFD). However, most of these methods are designed for grayscale images without any color information used. They are usually not suitable when the duplicated regions have little structure or have undergone various transforms. We propose a CMFD method using local geometrical color invariant features to detect duplicated regions. The method starts by calculating the color gradient of the inspected image. Then, we directly take the color gradient as the input for scale invariant features transform (SIFT) to extract color-SIFT descriptors. Finally, keypoints are matched and clustered before their geometrical relationship is estimated to expose the duplicated regions. We evaluate the detection performance and computational complexity of the proposed method together with several popular CMFD methods on a public database. Experimental results demonstrate the efficacy of the proposed method in detecting duplicated regions with various transforms and poor structure.

Illumination variability is one of the most important issues affecting imagery matching performances and still remains a critical problem in the literature, although different levels of improvement have been reported in recent years. This study proposes an illumination robust image matching method. There are three steps in the proposed method: first, local regions are extracted and matched from the input images by using a multiresolution region detector and an illumination robust shape descriptor; second, an algorithm is proposed to estimate the overlapping areas of images and enhance them based on the region matches; finally, general feature detectors and descriptors are combined to process the previous results for illumination robust matching. Experimental results demonstrate that the proposed matching method provides significant improvement in robustness for illumination change images compared with traditional methods.

Visual tracking is a fundamental task and has many applications in computer vision. We incorporate local dictionary and L0 regularized low-rank features into the particle filter framework to address this problem. Specifically, by developing an efficient L0 regularized sparse coding model to incrementally learn low-rank features for the tracking target and incorporating a local dictionary into low-rank features to build the observation model, we establish a robust online object tracking system. As a nontrivial byproduct, we also develop numerical algorithms to efficiently solve the resulting nonconvex optimization problems. Compared with conventional methods, which often directly use corrupted observations to form the dictionary, our low-rank feature-based dictionary successfully removes occlusions and exactly represents the intrinsic structure of the object. Furthermore, in contrast to the traditional holistic methods, the local strategy contains abundant partial and spatial information, thus enhancing the discrimination of our observation model. More importantly, the L0 norm-based hard sparse coding can successfully reduce the redundant information while preserving the intrinsic low-rank features of the target object, leading to a better appearance subspace updating scheme. Experimental results on challenging sequences show that our method consistently outperforms several state-of-the-art methods.

Face alignment is critical for face recognition, and the deep learning-based method shows promise for solving such issues, given that competitive results are achieved on benchmarks with additional benefits, such as dispensing with handcrafted features and initial shape. However, most existing deep learning-based approaches are complicated and quite time-consuming during training. We propose a compact face alignment method for fast training without decreasing its accuracy. Rectified linear unit is employed, which allows all networks approximately five times faster convergence than a tanh neuron. An eight learnable layer deep convolutional neural network (DCNN) based on local response normalization and a padding convolutional layer (PCL) is designed to provide reliable initial values during prediction. A model combination scheme is presented to further reduce errors, while showing that only two network architectures and hyperparameter selection procedures are required in our approach. A three-level cascaded system is ultimately built based on the DCNNs and model combination mode. Extensive experiments validate the effectiveness of our method and demonstrate comparable accuracy with state-of-the-art methods on BioID, labeled face parts in the wild, and Helen datasets.

We address the problem of fusing multifocus images based on the phase congruency (PC). PC provides a sharpness feature of a natural image. The focus measure (FM) is identified as strong PC near a distinctive image feature evaluated by the complex Gabor wavelet. The PC is more robust against noise than other FMs. The fusion image is obtained by a new fusion rule (FR), and the focused region is selected by the FR from one of the input images. Experimental results show that the proposed fusion scheme achieves the fusion performance of the state-of-the-art methods in terms of visual quality and quantitative evaluations.

A method to restore the depth map of an RGB-D image using smooth depth neighborhood (SDN) supports is presented. The SDN supports are computed based on the corresponding color image of the depth map. Compared with the most widely used square supports, the proposed SDN supports can well-capture the local structure of the object. Only pixels with similar depth values are allowed to be included in the support. We combine our SDN supports with the joint bilateral filter (JBF) to form the SDN-JBF and use it to restore depth maps. Experimental results show that our SDN-JBF can not only rectify the misaligned depth pixels but also preserve sharp depth discontinuities.

Accurate singular point (SP) detection is an important factor in fingerprint (FP) recognition systems. We propose an algorithm to detect SPs in FP images. Our idea is based on the observation that the orientation field (OF) at the regions containing SPs has high variation, whereas in the other regions, it is smooth. Thus, a pixel-wise descriptor that comprises orientation-deviation (OD)-based features is proposed to measure the OF variation in the local neighborhood of a pixel which we call OF energy. Candidate SPs are characterized by locations where the OF energy function has local gradual maxima. Furthermore, the OD-based descriptor exhibits some advanced topological properties, in particular the descriptor profile tendency, which are highly correlated with the SP type. These properties are used to filter out some spurious SPs. A second refining step based on an extended Poincaré index is then applied to keep only genuine SPs with their information. The proposed algorithm has the ability to accurately detect the classical singularities as well as the arch-type SP. Experiments conducted over the public databases FVC2002 db1 and db2 confirm its accuracy and reliability with a reduced false alarm rate in comparison to other proposed methods.

We present a multiexposure image fusion method that can enhance details, yet effectively improve brightness in the final result. In addition, a valid weight measurement is developed to remove motion objects in dynamic scenes. During the fusion process, each source image is first decomposed into one low-pass sub-band and a series of high-pass directional sub-bands using the nonsubsampled contourlet transform. Then the blended sub-bands are constructed by weight maps of the source images. To preserve the details of source images and adjust brightness of final image, gain control maps are used for each fused sub-band. Experimental results demonstrate that the proposed method significantly outperforms the traditional methods in terms of both visual inspection and objective evaluation, especially in cases which the regions of interest are in dark areas.

Image registration is an essential process for image fusion, and fusion performance can be used to evaluate registration accuracy. We propose a maximum likelihood (ML) approach to joint image registration and fusion instead of treating them as two independent processes in the conventional way. To improve the visual quality of a fused image, a gradient strength (GS) regularization is introduced in the cost function of ML. The GS of the fused image is controllable by setting the target GS value in the regularization term. This is useful because a larger target GS brings a clearer fused image and a smaller target GS makes the fused image smoother and thus restrains noise. Hence, the subjective quality of the fused image can be improved whether the source images are polluted by noise or not. We can obtain the fused image and registration parameters successively by minimizing the cost function using an iterative optimization method. Experimental results show that our method is effective with transformation, rotation, and scale parameters in the range of [−2.0, 2.0] pixel, [−1.1  deg, 1.1 deg], and [0.95, 1.05], respectively, and variances of noise smaller than 300. It also demonstrated that our method yields a more visual pleasing fused image and higher registration accuracy compared with a state-of-the-art algorithm.

We propose a new method for JPEG2000 region of interest (ROI) coding called the background pretruncated method (BPTM). An algorithm to calculate the key parameter in the BPTM is also proposed. The published JPEG2000 ROI coding methods have difficulty in balancing ROI quality and system computation. Methods based on shifting coefficients add much computation, and they significantly penalize the coding efficiency. Implicit methods cannot obtain as high-ROI quality as the coefficient shift methods. To efficiently achieve ROI coding, the BPTM assigns a proper number of the least significant bitplanes of the background to zero, and it can not only achieve high-ROI quality but also add no computation to the other modules in JPEG2000. Additionally, the BPTM supports arbitrary shaped ROI coding, and the codestream is compatible with JPEG2000 standard.

We propose a level set method for image segmentation which introduces the moment competition and weakly supervised information into the energy functional construction. Different from the region-based level set methods which use force competition, the moment competition is adopted to drive the contour evolution. Here, a so-called three-point labeling scheme is proposed to manually label three independent points (weakly supervised information) on the image. Then the intensity differences between the three points and the unlabeled pixels are used to construct the force arms for each image pixel. The corresponding force is generated from the global statistical information of a region-based method and weighted by the force arm. As a result, the moment can be constructed and incorporated into the energy functional to drive the evolving contour to approach the object boundary. In our method, the force arm can take full advantage of the three-point labeling scheme to constrain the moment competition. Additionally, the global statistical information and weakly supervised information are successfully integrated, which makes the proposed method more robust than traditional methods for initial contour placement and parameter setting. Experimental results with performance analysis also show the superiority of the proposed method on segmenting different types of complicated images, such as noisy images, three-phase images, images with intensity inhomogeneity, and texture images.

Various approaches have been proposed for video anomaly detection. Yet these approaches typically suffer from one or more limitations: they often characterize the pattern using its internal information, but ignore its external relationship which is important for local anomaly detection. Moreover, the high-dimensionality and the lack of robustness of pattern representation may lead to problems, including overfitting, increased computational cost and memory requirements, and high false alarm rate. We propose a video anomaly detection framework which relies on a heterogeneous representation to account for both the pattern’s internal information and external relationship. The internal information is characterized by slow features learned by slow feature analysis from low-level representations, and the external relationship is characterized by the spatial contextual distances. The heterogeneous representation is compact, robust, efficient, and discriminative for anomaly detection. Moreover, both the pattern’s internal information and external relationship can be taken into account in the proposed framework. Extensive experiments demonstrate the robustness and efficiency of our approach by comparison with the state-of-the-art approaches on the widely used benchmark datasets.

We focus on motion saliency detection, which has attracted much attention in recent years. Distinct from conventional algorithms, without considering the spatial information of the input frames, our method is solely based on the temporal difference of corresponding pixels. To be specific, the difference is modeled as two parts, i.e., symmetric frame difference and background sample difference. The first term calculates the differences of adjacent frames, including not only the previous n frames but also the backward n frames. To obtain the background sample difference, N samples taken from the previous frames are stored, and the differences between the coming pixel and all its corresponding samples are accumulated. The initialization and updating policies of the samples follow the off-the-shelf visual background extractor method. Finally, the motion saliency map is generated by fusing the aforementioned two features. We describe our method in full detail and compare it with three other state-of-the-art motion detection techniques. Experiments on a variety of visual surveillance databases verify that the proposed algorithm consistently outperformed the compared methods. We also demonstrate how the generated saliency maps can be used to create high-quality segmentation masks for moving object detection.

Poor visibility due to the effects of light absorption and scattering is challenging for processing underwater images. We propose an approach based on dehazing and color correction algorithms for underwater image enhancement. First, a simple dehazing algorithm is applied to remove the effects of haze in the underwater image. Second, color compensation, histogram equalization, saturation, and intensity stretching are used to improve contrast, brightness, color, and visibility of the underwater image. Furthermore, bilateral filtering is utilized to address the problem of the noise caused by the physical properties of the medium and the histogram equalization algorithm. In order to evaluate the performance of the proposed approach, we compared our results with six existing methods using the subjective technique, objective technique, and color cast tests. The results show that the proposed approach outperforms the six existing methods. The enhanced images, as a result of implementing the proposed approach, are characterized by relatively genuine color, increased contrast and brightness, reduced noise level, and better visibility.

Vanishing points (VPs) are crucial for inferring the three-dimensional structure of a scene and can be exploited in various computer vision applications. Previous VP detection algorithms have been proven effective but generally cannot guarantee a strong performance in both accuracy and computational time. We propose an artificial bee colony algorithm called dynamic clustering artificial bee colony (DCABC) that accurately and efficiently detects VPs in the image plane. The task is regarded as a dynamic line-clustering problem, and the line clusters are initialized by their orientation information. Inspired by the foraging behavior of bees, DCABC selects the clustering center and reclassifies the line segments based on a distance criterion until the terminating condition is met. The optimal line clusters determine the estimated VP. The dissimilarity among solutions is measured by the Hamming distance between two binary vectors, which simplifies the new solution construction. The performances of the proposed and existing algorithms are evaluated on the York Urban database. The results verify the efficiency and accuracy of our proposed algorithm.

The restoration of blurred images corrupted by Poisson noise is an important task in various applications such as medical imaging, microscopy imaging, and so on. We focus on mean curvature-based regularization to address the Poisson noise image restoration problem. Furthermore, we derive a numerical algorithm based on the augmented Lagrange multiplier method with a splitting technique. In order to simultaneously demonstrate the effectiveness of the proposed method for Poisson noise removal with deblurring, we conduct systematic experiments on both nature images and biological images. Experimental results show that the proposed approach can produce higher quality results and more natural images compared to some state-of-the-art variational algorithms recently developed.

Image reconstruction is a typical ill-posed inverse problem that has attracted increasing attention owing to its extensive use. To cope with the ill-posed nature of this problem, many regularizers have been presented to regularize the reconstruction process. One of the most popular regularizers in the literature is total variation (TV), known for its capability of preserving edges. However, TV-based reconstruction methods often tend to produce staircase-like artifacts since they favor piecewise constant solutions. To overcome this drawback, we propose to develop a second-order total generalized variation (TGV_α²)-based image reconstruction model with a combined L^1,2 data-fidelity term. The proposed model is applicable for restoration of blurred images with mixed Gaussian-impulse noise, and can be effectively used for undersampled magnetic resonance imaging. To further enhance the image reconstruction, a box constraint is incorporated into the proposed model by simply projecting all pixel values of the reconstructed image to lie in a certain interval (e.g., 0, 1 for normalized images and [0, 255] for 8-bit images). An optimization algorithm based on an alternating direction method of multipliers is developed to solve the proposed box-constrained image reconstruction model. Comprehensive numerical experiments have been conducted to compare our proposed method with some state-of-the-art reconstruction techniques. The experimental results have demonstrated its superior performance in terms of both quantitative evaluation and visual quality.

The attenuation of the light transmitted through air can reduce image quality when taking a photograph outdoors, especially in a hazy environment. Hazy images often lack sufficient information for image recognition systems to operate effectively. In order to solve the aforementioned problems, this study proposes a hybrid method combining fuzzy theory with weighted estimation for the removal of haze from images. A transmission map is first created based on fuzzy theory. According to the transmission map, the proposed method automatically finds the possible atmospheric lights and refines the atmospheric lights by mixing these candidates. Weighted estimation is then employed to generate a refined transmission map, which removes the halo artifact from around the sharp edges. Experimental results demonstrate the superiority of the proposed method over existing methods with regard to contrast, color depth, and the elimination of halo artifacts.

A scheme of data hiding directly in a partially encrypted version of H.264/AVC videos is proposed which includes three parts, i.e., selective encryption, data embedding and data extraction. Selective encryption is performed on context adaptive binary arithmetic coding (CABAC) bin-strings via stream ciphers. By careful selection of CABAC entropy coder syntax elements for selective encryption, the encrypted bitstream is format-compliant and has exactly the same bit rate. Then a data-hider embeds the additional data into partially encrypted H.264/AVC videos using a CABAC bin-string substitution technique without accessing the plaintext of the video content. Since bin-string substitution is carried out on those residual coefficients with approximately the same magnitude, the quality of the decrypted video is satisfactory. Video file size is strictly preserved even after data embedding. In order to adapt to different application scenarios, data extraction can be done either in the encrypted domain or in the decrypted domain. Experimental results have demonstrated the feasibility and efficiency of the proposed scheme.

In image-based three-dimensional (3-D) reconstruction, one topic of growing importance is how to quickly obtain a 3-D model from a large number of images. The retrieval of the correct and relevant images for the model poses a considerable technological challenge. The “image vocabulary tree” has been proposed as a method to search for similar images. However, a significant drawback of this approach is identified in its low time efficiency and barely satisfactory classification result. The method proposed is inspired by, and improves upon, some recent methods. Specifically, vocabulary quality is considered and multivocabulary trees are designed to improve the classification result. A marked improvement was, indeed, observed in our evaluation of the proposed method. To improve time efficiency, graphics processing unit (GPU) computer unified device architecture parallel computation is applied in the multivocabulary trees. The results of the experiments showed that the GPU was three to four times more efficient than the enumeration matching and CPU methods when the number of images is large. This paper presents a reliable reference method for the rapid construction of a free network to be used for the computing of 3-D information.