Video super-resolution (VSR) is an image restoration task, aiming to reconstruct a high-resolution (HR) video from its down-sampled low-resolution (LR) version. Convolutional neural networks (CNNs) have been applied to VSR successfully. Explicit motion estimation and motion compensation (ME&MC) module is commonly used in the previous CNNs-based methods to better exploit input frames’ temporal similarity. We proposed a VSR network without an explicit ME&MC module. Our network makes full use of spatiotemporal information and can implicitly capture motion relations between frames. Specifically, we proposed an enhanced deep feature extraction module (EDFEM) to extract deep features from input frames. EDFEM exploits not only intra-frame spatial information but also inter-frame temporal information to enhance feature representation. Furthermore, we proposed a residual up-down block (RUDB) to fuse features. RUDB adopts up- and down-sampling layers as the residual branch. Compared to the common residual block, RUDB addresses mutual dependencies of LR and HR images. Visual and quantitative results show that our method achieves state-of-the-art performance.