Risley prisms can serve as a beam steering device to move field of view and appear to be a promising solution to boresight adjustment in imaging tracking applications. It is desirable to deduce the inherent relation among target offsets, target’s line of sight (LOS) deviations from sensor’s boresight and the required prism rotations. The boresight adjustment model based on Risley prisms is described. Reverse ray tracing is performed and case examples are given to characterize the nonlinear relationship between target offset and its LOS deviation. A boresight adjustment method based on the inverse ray tracing and the two-step method is then illustrated. The simulation results show that the target’s LOS deviations from the boresight are stretched along the boresight’s radial direction relative to the target offsets. By the proposed boresight adjustment method, the required rotations of the prisms can be derived from the target offset. The singularity problem is also pointed out using this method, and the corresponding conclusions are consistent with those deduced in many works. The research demonstrates the inner mechanisms of the imaging tracking based on Risley prisms in principle. The proposed boresight adjustment method can provide guidance for controlling the circular motion of the prisms.