The accurate segmentation and counting of the reproductive organs within the herbarium specimen play an important role in studying the impact of climate change on plant development over time. Recently, the researchers have gained a lot of knowledge about plant phenology owing to herbaria’s digitization efforts, which may help accelerate plant phenology research by making large digitized specimen collections publicly available. Nevertheless, the automatic segmentation and counting of the reproductive organs is a challenging problem. This is because of the high variability of reproductive organs, which vary in size, shape, orientation, and color. The use of machine learning techniques, including deep learning, has recently been shown to be helpful in this endeavor. We proposed in this paper a deep learning method based on the refined Mask Scoring R-CNN approach to segment and count reproductive organs, including buds, flowers, and fruits from specimen images. Our proposed method achieved a precision rate of 94.5% and a recall rate of 93%.