The development of Convolutional Neural Networks (CNN) has expanded with the accelerated progress of IT, as well as with the needs of the autonomous vehicle (AV) implementation. The specifics and requirements of AV towards the infrastructure primarily relate to the condition and quality of traffic signs. For the independent participation of these vehicles in traffic, an impeccable traffic sign condition is required, which is often not the case in practice. Damaged, faded, obscured, or vandalized traffic signs can usually be seen in the road network, which can impede the movement of AV in traffic. In the existing literature, little or very little attention is focused on the problem of identifying and classifying damaged and especially vandalized traffic signs. In this paper, the mentioned problem is addressed, and the CNN model is proposed. This model has been tested on a specially designed novel and challenging database containing 6,000 real-time images of traffic signs in the road network of the Republic of Serbia. This model is invariant to different lighting and weather (nighttime and fog) conditions. In this case study, the model reached an overall accuracy of 99.17%, whereby all vandalized and damaged traffic signs are accurately identified and classified.