In this paper, we propose a diffusion-based approach to improve an input similarity metric. The diffusion process propagates similarity mass along the intrinsic manifold of data points. Our approach results in a global similarity metric which differs from the query-specific one for ranking produced by label propagation [26]. Unlike diffusion maps [7], our approach directly improves a given similarity metric without introducing any extra distance notions. We call our approach Self-Smoothing Operator (SSO). To demonstrate its wide applicability, experiments are reported on image retrieval, clustering, classification, and segmentation tasks. In most cases, using SSO results in significant performance gains over the original similarity metrics, with also very evident advantage over diffusion maps.