Most state-of-the-art single image blind deblurring techniques are still sensitive to image noise, leading to serious performance degradation in their blur kernel estimation when the input image noise increases. We found that reliable kernel estimation could not be given by directly using denoising and existing deblurring algorithms in many cases. We focus on how to estimate a good blur kernel from a noisy blurred image via using the image structure. First, we applied denoising as a preprocess to remove the input image noise and then computed salient image structure of the denoised result based on the total variation model. We also applied a gradient selection method to remove those salient edges that have a possible adverse effect on blur kernel estimation. Next, we adopted a two-phase estimation strategy to obtain higher quality blur kernel estimation by jointly applying kernel estimation from salient image structure and iterative support detection (ISD) kernel refinement. Finally, we used the nonblind deconvolution method based on sparse prior knowledge to restore the latent image. Extensive experiments testify to the superiority of the proposed method over state-of-the-art algorithms, both qualitatively and quantitatively.