Convolutional neural networks (CNN) have immensely promoted the development of face recognition (FR) technology. In order to achieve global accuracy, CNN models tend to be deeper or multiple local facial patch ensembles, leading to excessive amounts of calculation. We address these deep FR problems and propose a lightened deep learning framework under an open-set protocol to achieve a good classification effect and streamline the model itself. To this end, we improve the SphereFace that enables the deep network to learn angularly discriminative features more efficiently. First, global average pooling (GAP) is introduced to replace the original fully connected (FC) layer, which greatly reduces the storage of the model. Compared to the widely used FC layer, GAP can reduce the number of parameters and avoid overfitting. Then multilayer perceptron is added between convolution layers, which increases the ability to characterize features. These models are trained on the CASIA-WebFace dataset and evaluated on the Labeled Faces in the Wild and YTF datasets, which show the excellent performance of lightened SphereFace (L-SphereFace) in FR tasks. At the same time, computational cost is reduced in comparison with the released SphereFace model. The storage space of the model is also greatly compressed.