As a highly toxic heavy metal ion, divalent mercuric ion (Hg²⁺) is one of the most widely diffused and hazardous environmental pollutants. In this work, a simple, portable, and inexpensive fiber-optic sensor based on surface plasmon resonance (SPR) effect was developed for Hg²⁺ detection, which takes advantage of 4-mercaptopyridine (4-MPY)-functionalized Au nanoparticles (Au NPs/4-MPY) as a signal amplification tag. Based on the coordination between Hg²⁺ and nitrogen in the pyridine moiety, we developed the sensor by self-assembling 4-MPY on Au film surfaces to capture Hg²⁺ and then introducing Au NPs/4-MPY to generate a plasmonic coupling structure with the configuration of nanoparticle-on-mirror. The coupling between localized SPR increased changes in SPR wavelength, which allowed highly sensitive Hg²⁺ sensing in aqueous solution. The sensor exhibited superior selectivity for Hg²⁺ detection compared with other common metal ions in water. The sensor’s Hg²⁺ detection limit is 8 nM under optimal conditions. Furthermore, we validated the sensor’s practicality for Hg²⁺ detection in tap water samples and demonstrated its potential application for environmental water on-site monitoring.