Albeit offering many benefits, smartphones can pose a severe privacy threat to users. While some users might simply be not aware of privacy issues, others are highly motivated to protect their data, but lack the ability and knowledge to do so. We developed an Android-based application called "FoxIT", which provides users with several education modules as well as a static smartphone and app permission analysis to increase both, privacy awareness and knowledge of mobile users. We conducted a first evaluation of FoxIT in a two-week field study with 31 users and were able to show that use of FoxIT not only leads to increased privacy awareness, but also improves knowledge about privacy related topics. Participants also reported to have improved the privacy conditions on their smartphone, actively informed themselves about privacy related topics, and prompted others to protect their data after using FoxIT. Our results indicate that it might be a promising approach to improve mobile users' privacy behavior by raising awareness and providing background information about privacy related topics.