Speaker recognition technology has been widely used in home automation recently, which can provide identification for the user efficiently. However, speaker recognition technology brings security risks as well as convenience, especially the privacy disclosure of the user's voiceprint templates. This paper proposes an efficient and privacy-preserving speaker verification scheme for home automation devices, named SEASON. We modify the high-accuracy deepspeaker framework in terms of privacy, based on residual network. With SEASON, users can access the home automation devices securely. Detailed security analysis shows that SEASON can actually resist various security threats. We implement SEASON with real datasets and 48 real records from four testers. Experiments reveal that SEASON can achieve efficient and accurate verification.