Many post-stroke patients suffer varying degrees of hand function and fine motor skills impairment. Both passive and active hand rehabilitation training are beneficial in improving the strength and dexterity of the hands. However, hand rehabilitation programs should be prescribed based on an accurate assessment of hand function. In this paper, we propose a novel method for hand function assessment, which can accurately measure multiple joint angles of a hand simultaneously using a portable infrared based imaging device. Different from traditional assessment methods that are often based on a clinician's subjective observations and ordinal charts, this method provides an accurate, fast, and objective evaluation using infrared imaging sensors. Performance evaluation and benchmarking for the proposed measurement system were carried out using the correlation coefficient (CC) method, the root mean squared error, and the percentage residual difference method (PRD). A clinical trial involving 25 participants resulted in a higher correlation with CC of 0.9672 and PRD of 8.8%, which indicated that the developed assessment framework is compliant with multiple assessment standards such as Swanson impairment evaluation and Fugl–Meyer assessment. The new hand function assessment method can be used to replace traditional methods for fine hand function modeling and assessment in rehabilitation medicine and can also play an important role in precision post-stroke function analysis.