Prefabricated steel pipes play a crucial role in prefabricated buildings, and maintaining their surface integrity is crucial to ensuring the safety of these buildings. We propose a surface defect detection algorithm for prefabricated steel pipes, D-YOLOv7-tiny, based on YOLOv7-tiny, to address the challenges of high parameter count and large computational requirements associated with traditional algorithms, making it difficult to deploy at resource-constrained terminals. By incorporating the squeeze-and-excitation attention mechanism into the backbone network, D-YOLOv7-tiny effectively minimizes the impact of redundant information and improves the network’s ability to extract features. In addition, distribution shifting convolution is implemented as a replacement for a portion of traditional convolution in the original effective layer aggregation network module network. This exchange reduces the computational workload of the model without affecting its expressive power. Subsequently, the lightweight and ubiquitous content-aware reassembly of features upsampling operator improved the feature merging of the network. Finally, an attention-based dynamic head was adopted to enhance the model’s robustness while minimizing parameter counts. Compared with YOLOv7-tiny, the mAP of D-YOLOv7-tiny was enhanced by 1.9% through experiments conducted on self collected datasets. During this process, the number of parameters and computational complexity decreased by 7.6% and 39.4%, respectively. The results show that this method achieves lightweight and meets the requirements of practical engineering accuracy and real-time performance.