For an effective dietary assessment system, it is necessary to keep track of the amount of food consumed. Food recognition is the first step to calorie estimation, and image processing technique is useful to achieve this. With the use of food image classification, people can count the amount of food taken and control the calories taken, which helps to reduce the risk of serious health conditions like hypertension, chronic diseases, and heart disease. The nature of food is very diverse, which makes the food image classification task more challenging. Deep learning methods for image classification give more accurate and efficient results as compared to traditional methods. This research work focuses on classifying Gujarati food images as no efforts have been made till now to classify Gujarati food images. A new dataset named “Traditional Gujarati Food Images Dataset (TGFD)” has been created. The dataset contains 1764 images belonging to five food classes and famous food items in Gujarat. The experiments start by implementing transfer learning on models, namely VGG16, VGG19, Resnet50, Inceptionv3, and Alexnet. Fine-tuning has been implemented on all models in order to increase accuracy. After fine-tuning all the models, the maximum accuracy achieved was “89.36%” on the Inception v3 model, but the loss was very high. Certain parameters, like the number of convolutional layers, number of neurons in fully connected layers, number of filters, and filter size, directly affect the model's accuracy. Taking these parameters into consideration to improve accuracy and reduce loss, this research work proposes a model named “depth-restricted convolutional neural network (DRCNN)” which achieves “95.48%” accuracy, which is remarkable. The DRCNN model contains 482,069 parameters, which is 48 times less than the parameters of the Inceptionv3 model, and the validation loss is only 0.8041. Introducing batch normalization in the proposed model drastically improves performance with a lower number of parameters. DRCNN has been tested on an increasing number of classes in the dataset and on different types of food datasets. In both cases, the model performs outstandingly, proving its versatility.