In recent years, ensemble learning methods show great effectiveness in improving model performance in several applications. Ensemble techniques rely on the incorporation of multiple different models together to get one optimal model. The primary assumption of ensemble techniques is that the cooperation among various classifiers will probably compensate for the mistakes of a single classifier and consequently, the ensemble's general output prediction would be better than the prediction of a single classifier. A key issue in the combination of classifiers is the diversity among its members. In this paper, we utilized model averaging as an ensemble learning technique for offline Arabic handwritten word recognition to train three residual networks (ResNet18) models. We demonstrate improvements by incorporating diversity in output prediction by using distinct techniques of optimization. To validate the proposed method, experiments have been carried on the IFN/ENIT (v2.0ple) database which contains 32,492 handwritten Arabic words of 937 unique Arabic words.