Correlation filter (CF)-based trackers have demonstrated promising performance in the field of unmanned aerial vehicle (UAV) tracking, benefiting from their high computational efficiency. However, model degradation and the boundary effect remain two major obstacles for CF-based trackers. We present an innovative temporal regularized and weighted surrounding-aware CF for UAV object tracking. First, we considered the consistency of response between the current frame and the previous frame of an object in a time series, which effectively enhances the discriminability and generalization capability of CF-based trackers to abrupt appearance variations. Second, we developed an innovative weighted context learning strategy that is able to enlarge the visual field of trackers to cope with the boundary effect. Specifically, we extract weighted surrounding patches with the same size and shape of the object to empower the discriminating ability of the CF. Finally, to validate the effectiveness of our proposed method, substantial comparison experiments with other state-of-the-art algorithms were conducted on three UAV standard data sets. The experimental results demonstrate that the algorithm proposed has good robustness for occlusion, deformation, fast motion, and other challenging disturbances in complex scenes and effectively tracks the target on the platform of a UAV.