Tongue diagnosis is an important clinical examination in Traditional Chinese Medicine. As the first step of the diagnosis, the accuracy of tongue image segmentation directly affects the subsequent diagnosis. Recently, deep learning‐based methods have been applied for tongue image segmentation and achieve promising results. However, these methods usually work well on one dataset and degenerate significantly on different distributed datasets. To deal with this issue, we propose a framework named Iterative cross‐domain tongue segmentation in the study. First, we train a tongue image segmentation U‐Net model on the source dataset. Then, we propose a tongue assessment filter to select satisfying samples based on predictions of the U‐Net model from the target dataset. Following, we fine‐tune the model on the selected samples along with the source domain. Finally, we iterate between the filtering and the fine‐tuning steps until the model is converged. Experimental results on two tongue datasets show that our proposed method can improve the dice score on the target domain from 70.11% to 98.26%, as well as outperform state‐of‐the‐art comparing methods.