Compressed sensing (CS) shows significant potential in the field of active millimeter-wave (mmW) synthetic aperture radar (SAR) imaging due to the merits of reducing system complexity and achieving high-speed sensing. However, most CS-driven imaging methods suffer from the excessive computational burden, since the calculative steps always rely on vectorization and consequently lead to extremely large-scale matrix operations. To address this issue, we propose an efficient alternating direction method of multipliers (ADMMs) framework for mmW 3-D SAR imaging. In our scheme, we utilize the single-frequency holographic (SFH) technique and construct SFH-based forward/inverse sensing operators rather than converting the imaging process into a special case of “linear inverse problems,” by which the large-scale matrix inversions are avoided and consequently the computational complexity is reduced. Based on the SFH functional measurement model, the SFH-ADMM is derived to reconstruct the 3-D image from sparsely sampled measurement echo while suppressing noisy clutters and ambiguities. Besides, the SFH-ADMM iteration steps undergird a neural network design, yielding a tailored SFH-ADMM-Net with trainable parameters and layer-fixed structures, which further shorten the execution time and improve reconstruction performance. The network is trained by simulated data, which are generated according to the radar signal model. Extensive experiments, including simulations and laboratory tests, demonstrate the superiority of the proposed algorithms in terms of both reconstruction accuracy and computational speed.