In millimeter-wave imaging systems, the inversion efficiency of both the narrowband and broadband systems is affected by the target’s distribution distance. The mismatched distances cause inefficiency of the system, and the recovered images suffer from displacement blur. We analyze in detail the reasons for this phenomenon and propose a simple method to overcome it. Minimizing the sharpness metric of images is a key strategy to solve this problem. Using this method, the exact range information of the target can be obtained from the minimum sharpness of the unknown object, and then, the region of interest of the inversion can be located. Both synthetic and full-wave simulation data demonstrate the superiority of the method. Finally, the efficiency of the method is verified by the in-house developed test system.