Convolutional neural networks have been proven to have strong feature representation ability; however, they often require large training samples and high computation that are infeasible for real-time finger vein verification. To address this limitation, we propose a lightweight deep-learning framework for finger vein verification. First, we designed a lightweight two-channel network that has only three convolution layers for finger vein verification. Then, we extracted the mini-ROI from the original image to better solve the displacement problem based on the evaluation of the two-channel network. Finally, we present a two-stream network to integrate the original image and the mini-ROI that achieves results superior to the current state of the art on both the MMCBNU and SDUMLA databases.