We propose a contrast enhancement algorithm considering surrounding information by illumination image. Conventional contrast enhancement techniques can be classified as a retinex-based method and a tone mapping function-based method. However, many retinex methods suffer from high-computational costs or halo artifacts. To cope with these problems, efficient edge-preserving smoothing methods have been researched. Tone mapping function-based methods are limited in terms of enhancement since they are applied without considering surrounding information. To solve these problems, we estimate an illumination image with local adaptive smoothness, and then utilize it as surrounding information. The local adaptive smoothness is calculated by using illumination image properties and an edge-adaptive filter based on the just noticeable difference model. Additionally, we employ a resizing method instead of a blur kernel to reduce the computational cost of illumination estimation. The estimated illumination image is incorporated with the tone mapping function to address the limitations of the tone mapping function-based method. With this approach, the amount of local contrast enhancement is increased. Experimental results show that the proposed algorithm enhances both global and local contrasts and produces better performance in objective evaluation metrics while preventing a halo artifact.