Identifying tweets related to infrastructure damage during a crisis event is an important problem. However, the unavailability of labeled data during the early stages of a crisis event poses major challenge in training suitable models. Several domain adaptation strategies have been proposed for text classification that can be used to train models using available source data of previous crisis events and apply on a target data related to a current event. However, these approaches are insufficient to handle the distribution drift in the source and target data along with the class imbalance in the target data. In this paper we introduce an Ensemble learning approach with a Decoupled Adversarial (EnDeA) model to classify infrastructure damage tweets in a target tweet dataset. EnDeA is an ensemble of three different models two of which separately learn the event invariant and specific features of a target data from a set of source and target data. The third model which is an adversarial model helps to improve the prediction accuracy of both models. Unlike the existing approaches that also identify the domain invariant and specific properties of target data for sentiment classification, our method works for short texts and can better handle the distribution drift and class imbalance problem. We rigorously investigate the performance of the proposed approach using multiple public datasets and compare it with several state-of-the-art baselines. We discover that EnDeA outperforms these baselines with around 20% improvement in the 1 scores.