Imaging through refractive media is challenged by the alteration of light trajectory from their linear path, thus incurring severe distortions. These distortions are depth-dependent, non-linear, and, more importantly, alter the standard single viewpoint geometry into an axial form. Though modeling such imaging systems has been studied in the past, an in-depth analysis of their two-view geometry is still lacking. We demonstrate in this paper that the value of such form is not limited to the estimation of the relative motion as it allows to calibrate the system independent of any scene knowledge. Therefore, we relieve the establishment of external reference frames and estimate the system parameters through the course of the mission and on-the-fly. In our development, we identify relations among the parameters that allow us to separate them into two orthogonal groups. Using these relations we calibrate the system and solve the pose components. Only a minimal set of five corresponding points is needed for that purpose. We also show that while existing solutions constrain subsets of the parameters or apply hardware installations, our model requires no additional information, still estimating all parameters simultaneously and directly. In the paper, the formulation of two common imaging setups are developed, the first is of a single camera acquiring data through motion, and the other is of a stereo-rig, where the relative pose is fixed but where the system model is more involved. Numerical and experimental results demonstrate that the proposed calibration model exhibits stability to the presence of high levels of noise and achieves high levels of accuracy. Furthermore, we demonstrate how our modeling improves the accuracy by an order of magnitude or more compared to the state-of-the-art.