In this paper, we describe a deep convolutional neural network (DNN) model trained with simulated breast diffuse optical tomography data with realistic noise characteristics to solve the inverse problem in a fast single-pass feed forward reconstruction. In addition to an AUTOMAP-inspired network structure, our DNN model, a.k.a. FDU-Net, is also comprised of a U-Net to further improve the image quality. We demonstrate that our FDU-Net model can successfully recover nearly the full contrast of inclusions with accurate localization at millisecond-scale speed, outperforming the conventional finite element-based (FEM) methods. Trained with cases with a single spherical inclusion, the FDU-Net model can also recover multi-inclusions and irregular-shaped cases, demonstrating advantages of generalization.