Object tracking technology is being continually improved to ensure robust tracking performance under real conditions, such as cluttered background, fast motion, and lengthy occlusions. The performance of these video trackers depends significantly on evaluation methodologies adopted during the assessment of tracking algorithms. Most of these video trackers are evaluated using video sequences annotated with the simulated visual attributes of objects. Generating such sequences with the requisite attributes is laborious and time consuming. These annotated video sequences may fail to address real challenges; hence, the trackers must be tested using available opportunity targets in lengthy field trials. We present an innovative scheme for the hardware-in-loop testing of such video trackers for efficient evaluation under complex background scenarios. The designed simulator can provide the real-time visual characteristics of object parameters with a live background without halting the evaluation process. The proposed method facilitates the efficient evaluation of a closed-loop video tracker by providing a virtually endless video sequence containing realistic visual attributes of potential objects. In addition, the validation of the tracker’s performance in realistic scenarios, the comparative performance of tracking algorithms under varying operational conditions, and an in-depth analysis of the tracker system parameters are presented.