The latest high-efficiency video coding (HEVC) standard adopts a recursive quad-tree block structure with the coding-unit (CU) size variable depending on video content. It substantially improves the coding efficiency and also dramatically increases complexity. Therefore, a fast CU size selection algorithm based on hierarchical quad-tree correlations (HQTCs) is proposed. First, for each coding tree unit, the partition information at each depth is recorded in a table that reflects the appearance of the quad-tree structure. Then, by using two techniques called top omitting and bottom pruning, the size of the current CU can be determined according to the subtree distributions of adjacent CUs instead of traversing all the depths. Additionally, a gray level co-occurrence matrix-based method is also introduced to further speedup the searching process. Experimental results show that the proposed algorithm can achieve on average a 26% computational time reduction under all configurations with a negligible BD-rate (Bjøntegaard Delta bitrate) increase of 0.47% compared with the original encoding scheme in HEVC test model HM13.0.