A new method for unsupervised segmentation of moving objects in infrared videos is presented. This method consists of two steps: difference image quantization and spatial segmentation. In the first step, the changed pixels in the difference image are quantized to several classes by using Bayes decision. It can be used to cluster the changed pixels belonging to the same moving object together. The pixels of the difference image are replaced by their corresponding class labels, thus forming a class-map of the difference image. In the second step, each class in the class-map is considered as a subset of the possible seeds of moving objects. A self-adaptive region growing method is then used to image segmentation on the basis of these different subsets. One of the focuses of this work is on spatial segmentation, where a criterion is proposed for evaluation of moving object segmentation without ground truth in infrared videos. This criterion is used to evaluate the performance of the segmentation masks grown from different subsets of the possible seeds. The best segmented image is determined to be the final segmentation result. Experiments show the advantage and robustness of the proposed algorithm on real infrared videos.