In the field of fisheries, detecting the distribution of fish underwater is an important task for achieving accurate bait feeding. However, the current deep neural networks for fish detection are significantly more computationally intensive than previous methods due to their increased network depths. Additionally, drawbacks such as the difficulty of balancing accuracy and real-time performance limit the deployment of these algorithms in fishery end devices. To address this problem, this paper proposes an improved You Only Look Once version 5 (YOLOv5)-based underwater fish detection method called RC_YOLOv5. First, the Res2Net residual structure is introduced to represent multiscale features at a finer granularity and increase the perceptual field of the network while reducing the computational power of the model. Second, a coordinate attention mechanism is introduced to suppress the interference of the background and help the network locate its target more accurately. Finally, coordinate attention is embedded into the tail of Res2Net to form a residual attention structure, and this structure is used to replace the original bottleneck structure in the YOLOv5 model to improve its accuracy. Experiments show that the proposed model has good performance on a self-built fish dataset, reaching 95.7% and 95.4% precision and mean average precision (mAP), respectively. Compared with those of the original model, the precision of the proposed approach improves by 1.6%, the mAP improves by 0.6%, the number of computations is reduced by 22.2%, the model size is reduced by 23.5%, the detection rate reaches 263 frames per second (FPS) and the performance is better than that of other mainstream detection models. This method enables accurate and rapid fish detection in fisheries.