Multiple reference frame motion estimation (MRF-ME) is one of the most crucial tools in H.264/AVC to improve coding efficiency. However, it disciplines an encoder by giving extra computational complexity. The required computation proportionally expands when the number of reference frames used for motion estimation increases. Aiming to reduce the computational complexity of the encoder, various motion vector (MV) composition algorithms for MRF-ME have been proposed. However, these algorithms only perform well in a limited range of reference frames. The performance deteriorates when motion vector composition is processed from the current frame to a distant reference frame. In this paper, a reliable tracking mechanism for MV composition is proposed by utilizing only the relevant areas in the target macroblock and taking different paths through a novel selection process from a set of candidate motion vectors. The proposed algorithm is especially suited for temporally remote reference frames in MRF-ME. Experimental results show that compared with the existing MV composition algorithms, the proposed one can deliver a remarkable improvement on the rate-distortion performance with similar computational complexity.