Recent advancement of deep learning research has made significant impact on Natural Language Processing (NLP). However, many research challenges remain, such as effectively designing deep neural networks to better represent and understand semantics, which is essential for many NLP tasks. In this dissertation, we developed new Deep Neural Network architectures and applied them to three NLP tasks involving short texts: topic modeling, narrative quality evaluation, and text simplification. We first showed word embedding obtained from neural networks could improve the performance of topic modeling. Then, we proposed three innovative neural network readers that model textual chunks and their interrelations to understand semantics and evaluate the quality of short stories. Finally, we designed feature-rich sequence-to-sequence neural networks to automatically simplify complex text. The progress in each of the three tasks contributes significantly to representation and analysis of semantics of short texts. In empirical study, our approaches achieved the state-of-the-art performance using multiple real-world corpora.