Anomaly detection/prediction is the first step to secure IoT systems. It usually relies on wide domain knowledge to build up the tools to automatically detect/predict abnormal events or behaviors of an IoT system. However, an IoT system may consist of machines with different capabilities, functionalities and ages. Furthermore, abnormal events or behaviors are usually rare events. It is time-consuming and high-cost to build up the domain knowhow of the IoT systems and collect enough data points of the anomaly. In this article, we first identify the issues and challenges. Then we illustrate a general environment for anomaly detection/prediction on the IoT systems. Then we survey the core technologies and existing solutions that may be applied for anomaly detection/prediction. We also identify what cannot be achieved by the existing solutions. Then considering four datasets, we show the performance comparison for different solutions by running experiments.