Real-time vision-based system of fault detection (RVBS-FD) for freight trains aims to complete routine maintenance tasks efficiently for ensuring railway transportation security. However, most existing systems are designed to detect only one specific type of faults or even one fault, which fail to deal with multifault detection. Recently, the rapid development in deep learning techniques enables systems to provide a robust solution for the RVBS-FD of freight trains. However, general convolutional neural networks (CNNs) cannot fully meet the actual requirements in terms of the real time, accuracy, and resource constrains for the RVBS-FD of freight trains. To solve these problems, we propose a CNN-based detector called Light fault detection network for freight train images (FTI-FDNet) for the RVBS-FD of the freight train. First, we use the multiregion proposal networks (MRPN), which extract a set of prior bounding boxes to achieve initial fault proposal generation. Then, a powerful multilevel region-of-interest (RoI) pooling is presented for proposal classification and accurate detection. We finally design a reliable model reduction scheme (MRS) to pursue fast speed with high detection accuracy in a simple manner. The experimental results on five typical fault benchmarks indicate that our Light FTI-FDNet achieves higher accuracy and fast speed with about 17% model size of the well-known faster region-based CNN (R-CNN) detector, substantially outperforming the state-of-the-art methods.