The 5G cellular network employs non-orthogonal multiple access (NOMA) to enhance network connectivity and capacity, and device-to-device (D2D) communications to improve spectrum efficiency. However, the underlay D2D communications may destroy the execution condition for the successive interference cancellation (SIC) decoding of NOMA cellular networks by introducing the extra interference, which degrades the cellular transmission reliability. Thus, we develop the interlay mode as a special D2D mode for NOMA system, which enables the power domain multiplexing of the D2D pair and cellular users to eliminate the strong interference between them by the SIC decoding. When D2D pair conducts the selection between the interlay mode and underlay mode, the SIC decoding constraint should be satisfied at both D2D receiver and NOMA base station. In order to maximize the system sum rate while meeting the SIC decoding constraint, we propose a joint D2D mode selection and resource allocation scheme with interlay mode, which can be formulated as a combinatorial optimization problem. To tackle the combinatorial nature of mode selection and spectrum assignment, we first prove that the original problem can be reformulated as a maximum weight clique problem, and then propose a graph-based algorithm by applying branch-and-bound method to obtain its optimal solution. Finally, simulation results are provided to demonstrate that the interlay mode along with the proposed algorithms can coordinate D2D communications and NOMA cellular network to significantly improve the system sum rate and the D2D access rate.