Water is ubiquitous in natural systems, ranging from the vast oceans to the nanocapillaries in the earth crust or cellular organelles. In bulk or in intimate contact with solid surfaces, water molecules arrange themselves according to their hydrogen (H) bonding, which critically affects their short- and long-range molecular structures. Formation of H-bonds among water molecules designates the energy levels of certain nonbonding molecular orbitals of water, which are quantifiable by spectroscopic techniques. While the molecular architecture of water in nanoenclosures is of particular interest to both science and industry, it requires fine spectroscopic probes with nanometer spatial resolution and sub-eV energy sensitivity. Graphene liquid cells (GLCs), which feature opposing closely spaced sheets of hydrophobic graphene, facilitate high-resolution transmission electron microscopy (TEM) and electron energy-loss spectroscopy (EELS) measurements of attoliter water volumes encapsulated tightly in the GLC nanovessels. We perform in situ TEM and EELS analysis of water encased in thin GLCs exposed to room and cryogenic temperatures to examine the nanoscale arrangement of the contained water molecules. Simultaneous quantification of GLC thickness leads to the conclusion that H-bonding strengthens under increased water confinement. The present results demonstrate the feasibility of nanoscale chemical characterization of aqueous fluids trapped in GLC nanovessels and offer insights on water molecule arrangement under high-confinement conditions.