We present an adaptive object segmentation based on scene change detection techniques. First, the algorithm adaptively adjusts to skip the number of frames to cope with the amount of motion displacement to cover the entire object shape. Next, the spatial processing consists of noise removal and boundary smoothing techniques to discard background content as well as to remove the background noise, to obtain a segmented object. To evaluate the quality of segmentation, both of the standard benchmarks and camera imaging are employed. Results show that the proposed algorithm can achieve low error ratios under various standard sequence testing. The segmentation algorithm combines with MPEG-4 coding for camera imaging of video surveillance. This system can successfully demonstrate real-time video surveillance, as the frame rate is 10 to 30 using CPU-based software implementation.