Variable transform sizes have been adopted by the emerging international standard of high-efficiency video coding (HEVC). The HEVC test model (HM) supports four different transform sizes in the range of to . The transform size is chosen based on the rate-distortion optimization techniques. Variable transform sizes have a coding performance superior to the traditional fixed block-size transform. Meanwhile, this causes high computational complexity at the encoders. We propose a fast transform size decision algorithm incorporating three methods: (1) transform bypass based on the SKIP mode detection, (2) content-based transform size decision, and (3) early termination of variable-sized transform. The proposed algorithm first checks coding information from nearby spatially and temporally coding units (CUs) or treeblocks and the parent CU in the upper depth level. Based on the coding information from nearby CUs, only a small number of transform sizes are selected in the procedure of transform size decision. Simulation results show that our algorithm can reduce the complexity of the transform procession by more than 55% with a negligible loss of coding efficiency.