Image representation and classification are two fundamental tasks toward version understanding. Shape and texture provide two key features for visual representation and have been widely exploited in a number of successful local descriptors, e.g., scale invariant feature transform (SIFT), local binary pattern descriptor, and histogram of oriented gradient. Unlike these gradient-based descriptors, this paper presents a simple yet efficient local descriptor, named local color contrastive descriptor (LCCD), which captures the contrastive aspects among local regions or color channels for image representation. LCCD is partly inspired by the neural science facts that color contrast plays important roles in visual perception and there exist strong linkages between color and shape. We leverage -divergence as a robust measure to estimate the contrastive features between different spatial locations and multiple channels. Our descriptor enriches local image representation with both color and contrast information. Due to that LCCD does not explore any gradient information, individual LCCD does not yield strong performance. But we verified experimentally that LCCD can compensate strongly SIFT. Extensive experimental results on image classification show that our descriptor improves the performance of SIFT substantially by combination on three challenging benchmarks, including MIT Indoor-67 database, SUN397, and PASCAL VOC 2007.