We propose a robust local tracker based on red-green-blue (RGB) color channel fusion. In this tracker, the object is first divided into some patches with overlap region, and then the local sparse optimization from RGB color channels in each patch is solved via the mixed-norm regularization, which can realize the fusion of multicolor channel information. In the calculation of candidate object confidence, the confidences from RGB color channels are fused to obtain the total confidence of candidate object, this can accurately select the best candidate object. In the update module of template set and dictionary database, we design an adaptive update mechanism. The template and dictionary to delete are determined by sorting the cosine similarities between the tracking result and templates while the update of dictionary database is completed by replacing the old dictionaries with the reconstruction results of all the patches corresponding to the tracking result. This update method can effectively adapt to the appearance change of the object, and it can alleviate the tracking drift. Both qualitative and quantitative evaluations on challenging video sequences demonstrate that the tracker proposed is reliable and effective. It performs favorably against several state-of-the-art methods.