Using in situ optical spectroscopy we have investigated the temperature stability of the mixed atomic and molecular phases IV of dense deuterium and hydrogen. Through a series of low-temperature experiments at high pressures, we observe phase III-to-IV transformation, imposing constraints on the P-T phase diagrams. The spectral features of the phase IV-III transition and differences in appearances of the isotopes Raman spectra strongly indicate the presence of proton tunneling in phase IV. No differences between isotopes were observed in absorption spectroscopic studies, resulting in identical values for the band gap. The extrapolation of the combined band gap yields 375 GPa as the minimum transition pressure to the metallic state of hydrogen (deuterium). The minute changes in optical spectra above 275 GPa might suggest the presence of a new solid modification of hydrogen (deuterium), closely related structurally to phase IV.