Thanks to the rapidly developing technology, unmanned aerial vehicles (UAVs) are able to complete a number of tasks in cooperation with each other without need for human intervention. In recent years, UAVs, which are widely utilized in military missions, have begun to be deployed in civilian applications and mostly for commercial purposes. With their growing numbers and range of applications, UAVs are becoming more and more popular; on the other hand, they are also the target of various threats which can exploit various vulnerabilities of UAV systems in order to cause destructive effects. It is therefore critical that security is ensured for UAVs and the networks that provide communication between UAVs. In this survey, we aimed to present a comprehensive detailed approach to security by classifying possible attacks against UAVs and flying ad hoc networks (FANETs). We classified the security threats into four major categories that make up the basic structure of UAVs; hardware attacks, software attacks, sensor attacks, and communication attacks. In addition, countermeasures against these attacks are presented in separate groups as prevention and detection. In particular, we focus on the security of FANETs, which face significant security challenges due to their characteristics and are also vulnerable to insider attacks. Therefore, this survey presents a review of the security fundamentals for FANETs, and also four different routing attacks against FANETs are simulated with realistic parameters and then analyzed. Finally, limitations and open issues are also discussed to direct future work