As Internet of Things (IoT) involvement increases in our daily lives, several security and privacy concerns like linkability, unauthorized conversations, and side-channel attacks are raised. If they are left untouched, such issues may threaten the existence of IoT. They derive from two main reasons. One is that IoT objects are equipped with limited capabilities in terms of computation power, memory, and bandwidth which hamper the direct implementation of traditional Internet security techniques. The other reason is the absence of widely-accepted IoT security and privacy guidelines and their appropriate implementation techniques. Such guidelines and techniques would greatly assist IoT stakeholders like developers and manufacturers, paving the road for building secure IoT systems from the start and, thus, reinforcing IoT security and privacy by design. In order to contribute to such objective, we first briefly discuss the primary IoT security goals and recognize IoT stakeholders. Second, we propose a comprehensive list of IoT security and privacy guidelines for the edge nodes and communication levels of IoT reference architecture. Furthermore, we point out the IoT stakeholders such as customers and manufacturers who will benefit most from these guidelines. Moreover, we identify a set of implementation techniques by which such guidelines can be accomplished, and possible attacks against previously-mentioned levels can be alleviated. Third, we discuss the challenges of IoT security and privacy guidelines, and we briefly discuss digital rights management in IoT. Finally, through this survey, we suggest several open issues that require further investigation in the future. To the best of the authors’ knowledge, this work is the first survey that covers the above-mentioned objectives.