Image-based object detection provides a valuable basis for site information retrieval and construction progress monitoring. Machine learning approaches, such as neural networks, are able to provide reliable detection rates. However, labeling of training data is a tedious and time-consuming process, as it must be performed manually for a substantial number of images. The paper presents a novel method for automatically labeling construction images based on the combination of 4D Building Information Models and an inverse photogrammetry approach. For the reconstruction of point clouds, which are often used for progress monitoring, a large number of pictures are taken from the site. By aligning the Building Information Model and the resulting point cloud, it is possible to project any building element of the BIM model into the acquired pictures. This allows for automated labeling as the semantic information of the element type is provided by the BIM model and can be associated with the respective regions. The labeled data can subsequently be used to train an image-based neural network. Since the exact regions for all elements are defined, labels can be generated for basic tasks like classification as well as more complex tasks like semantic segmentation. To prove the feasibility of the developed methods, the labeling procedure is applied to several real-world construction sites, providing over 30,000 automatically labeled elements. The correctness of the assigned labels has been validated by pixel based area comparison against manual labels.