We propose a two-phase approach to restore images corrupted by impulsive noise based on sparse representation. In the first phase, we identify the outlier candidates—the pixels that are likely to be corrupted by impulsive noise. In the second phase, the image is denoised via dictionary learning by using the outlier-free data. The dictionary learning task is formulated as a modified minimization objective and solved under the alternating direction method. The experimental results demonstrate that our method can obtain better performances in terms of both quantitative evaluation and visual quality than the state-of-the-art impulse denoising methods.