Multi-modal medical image fusion, as a powerful tool for the clinical applications, has developed with the advent of various imaging modalities in medical imaging. The main motivation is to capture most relevant information from sources into a single output, which plays an important role in medical diagnosis. In this paper, a novel framework for medical image fusion based on framelet transform is proposed considering the characteristics of human visual system (HVS). The core idea behind the proposed framework is to decompose all source images by the framelet transform. Two different HVS inspired fusion rules are proposed for combining the low- and high-frequency coefficients respectively. The former is based on the visibility measure, and the latter is based on the texture information. Finally, the fused image is constructed by the inverse framelet transform with all composite coefficients. Experimental results highlight the expediency and suitability of the proposed framework. The efficiency of the proposed method is demonstrated by the different experiments on different multi-modal medical images. Further, the enhanced performance of the proposed framework is understood from the comparison with existing algorithms.