Device-to-device (D2D) communication is one of the key enabling technologies for 5G to support many connections efficiently. In a cooperative out-band D2D communication network, relay nodes have an essential role in receiving and forwarding information signals from source (S) to destination (D), making the system more reliable and with higher energy efficiency. Therefore, the relay selection method is very important in determining the best relay among existing relays to achieve maximum performance. This paper proposes a new Adaptive Relay Selection (ARS) scheme for cooperative out-band D2D networks based on channel gain value and transmission link distance. Firstly, the best relay is adaptively selected among N available relays (R) based on the maximum channel gain values between S to R if the channel gain-based signal-to-noise ratio (SNR) is greater than the distance-based SNR. Otherwise, the best relay is selected based on the minimum distance between S to R. We also analyzed an exact closed-form throughput and the total energy consumption required for a cooperative out-band D2D communication system using the Quantization-and-Forward (QF) protocol. The numerical results show that the proposed ARS scheme has a higher throughput than the two previous schemes: maximum channel gain and minimum distance. Furthermore, the proposed ARS can reduce the total energy consumption, which indirectly impacts the resulting energy efficiency level. The proposed ARS scheme achieves higher energy efficiency than the previous schemes, either by link distance or power allocation. So, the proposed ARS scheme for an out-band D2D communication network is an appropriate solution for the next generation of cellular communications.