Carotid atherosclerosis is one of the leading causes of cardiovascular disease with high mortality. Multi-contrast MRI can identify atherosclerotic plaque components with high sensitivity and specificity. Accurate segmentation of the diseased carotid artery from MR images is very essential to quantitatively evaluate the state of atherosclerosis. However, due to the complex morphology of atherosclerosis plaques and the lack of well-annotated data, the segmentation of lumen and wall is very challenging. Different from popular deep learning methods, in this paper, we propose an integration segmentation framework by introducing a lightweight prediction model and improved optimal surface graph cuts (OSG), which adopts a simplified flow line sampling and post-reconstructing method to reduce the cost of graph construction. Moreover, a flexibly adaptive smoothing penalty is presented for maintaining the shape of diseased carotid surface. For the experiments, we have collected an MR image dataset from patients with carotid atherosclerosis and evaluated our method by cross-validation. It can reach 89.68%/80.29% of dice coefficients and 0.2480 mm/0.3396 mm of average surface distances on the lumen/wall segmentation, respectively. The experimental results show that our method can generate precise and reliable segmentation of both lumen and wall of diseased carotid artery with a quite small training cost.