In recent years, curvilinear mask technology has emerged as a next-generation resolution enhancement method for photomasks, providing optimal margins by maximizing the degree of freedom in pattern design. However, this technology presents challenges in defining the layout design rule limits based solely on geometric information, such as width, space, and corner-to-corner. With the introduction of multi-beam mask writers for curvilinear pattern production, a distinct set of defects that are difficult to pre-detect by conventional mask rule check have occurred, as these cannot be explained by geometry terms alone. In this study, we propose a deep learning-based mask check method, named mask deep check (MDC) for pre-detect defects in inspection. The proposed vector graphics transformer (VGT) uses the state-of-the-art transformer architecture, drawing an analogy between the vertices of curvilinear patterns and words in natural language. We demonstrate improved performance of VGT-based MDC compared to a traditional rule-based approach and a convolutional neural network-based MDC method. Importantly, VGT exhibits robustness in recall, ensuring that defective patterns are not misclassified as normal, thereby preventing missed defects. Moreover, by employing attention maps to visualize VGT results, we gain explainability and reveal that mask defects may arise from issues related to the fabrication of specific designs, rather than being solely attributable to geometric features. VGT-based MDC contributes to a better understanding of the challenges associated with curvilinear mask technology and offers an effective solution for detecting mask defects.