The growth of the Internet of Things (IoT) presents challenges in the field of security. The Intrusion Detection System is an alternative to protecting the internet of things. In this study, we propose an intrusion detection system model that combines unsupervised algorithm and a deep neural network. Autoencoder as unsupervised learning algorithm has a function as a feature extractor that speeds up the learning process on a deep neural network. The performance of a deep learning model depends heavily on the selection of hyperparameters of neural network architecture. In this case, we used Bayesian Hyperparameter Optimization to perform hyperparameter tuning of deep learning models with various activation and weight initialization techniques. The accumulation result is useful to help determine the correct activation function and weight initialization and the hyperparameters that most influence the deep learning model. The results of this study show that Bayesian hyperparameter optimization can improve classification results significantly. Evaluation using the BoT-IoT dataset, the classification accuracy results in deep learning model can reach 99.99%.