In this review, we are pitting two theories against each other: the more accepted theory, the number sense theory, suggesting that a sense of number is innate and non-symbolic numerosity is being processed independently of continuous magnitudes (e.g., size, area, and density); and the newly emerging theory suggesting that (1) both numerosities and continuous magnitudes are processed holistically when comparing numerosities and (2) a sense of number might not be innate. In the first part of this review, we discuss the number sense theory. Against this background, we demonstrate how the natural correlation between numerosities and continuous magnitudes makes it nearly impossible to study non-symbolic numerosity processing in isolation from continuous magnitudes, and therefore, the results of behavioral and imaging studies with infants, adults, and animals can be explained, at least in part, by relying on continuous magnitudes. In the second part, we explain the sense of magnitude theory and review studies that directly demonstrate that continuous magnitudes are more automatic and basic than numerosities. Finally, we present outstanding questions. Our conclusion is that there is not enough convincing evidence to support the number sense theory anymore. Therefore, we encourage researchers not to assume that number sense is simply innate, but to put this hypothesis to the test and consider whether such an assumption is even testable in the light of the correlation of numerosity and continuous magnitudes.