Gesture recognition has become an important research in human-machine interface. Currently, most of gesture recognition schemes are based on vision sensor, which is sensitive to application environment. In contrast, millimeter wave sensor demonstrates better reliability to different environments. However, they normally deliver a less dense image and challenge high quality recognition. To address this issue, we propose a RD-T network-based gesture recognition method leveraging 4-dimension sensing capability of millimeter wave radar. By categorizing sensed images into Range-Time-Map (RTM), Doppler-Time-Map (DTM) and Range-Doppler-Map (RDM), we can perform high fidelity feature extraction and classification. In turn, an RD-T network has been designed to realize the branch network structure of the scheme. Experiments show that the achieved accuracy of RD-T network is 2% ~ 3% higher than a single parameter network, suggest the validity of the proposed high-dimensional feature fusion scheme.