The fine grained energy spectrum of quantum chaotic systems is widely believed to be described by random matrix statistics. A basic scale in such a system is the energy range over which this behavior persists. We define the corresponding time scale by the time at which the linearly growing ramp region in the spectral form factor begins. We call this time t ramp. The purpose of this paper is to study this scale in many-body quantum systems that display strong chaos, sometimes called scrambling systems. We focus on randomly coupled qubit systems, both local and k-local (all-to-all interactions) and the Sachdev-Ye-Kitaev (SYK) model. Using numerical results, analytic estimates for random quantum circuits, and a heuristic analysis of Hamiltonian systems we find the following results. For geometrically local systems with a conservation law we find t ramp is determined by the diffusion time across the system, order N 2 for a 1D chain of N qubits. This is analogous to the behavior found for local one-body chaotic systems. For a k-local system like SYK the time is order log N but with a different prefactor and a different mechanism than the scrambling time. In the absence of any conservation laws, as in a generic random quantum circuit, we find t ramp∼ log N, independent of connectivity.