Information on which weed species are present within agricultural fields is a prerequisite when using robots for site‐specific weed management. This study proposes a method of improving robustness in shape‐based classifying of seedlings toward natural shape variations within each plant species. To do so, leaves are separated from plants and classified individually together with the classification of the whole plant. The classification is based on common, rotation‐invariant features. Based on previous classifications of leaves and plants, confidence in correct assignment is created for the plants and leaves, and this confidence is used to determine the species of the plant. By using this approach, the classification accuracy of eight plants species at early growth stages is increased from 93.9% to 96.3%.