We propose a 16-ary quadrature amplitude modulation (16-QAM) signal decision algorithm termed as the joint support vector machine (SVM) and K-nearest neighbor (KNN) module (JSKM). The M-ary SVM makes the first decision for all the received symbols. Subsequently, the KNN algorithm makes the second decision for the received symbols inside the decision boundary of the SVM. In this manner, the advantages of the SVM can be exploited and the KNN algorithm can help overcome the limitations of the SVM. The proposed algorithm was implemented to perform the simulation of a 112-Gbps 16-QAM coherent optical transmission system. The results demonstrated the superiority of the JSKM algorithm as it could increase the transmission distance by 440 and 227 km compared to those obtained using the KNN and M-ary SVM algorithms, respectively.