Nowadays, in many companies, defect classification plays a vital role in surface quality measuring instruments. However, there is a conflict between accuracy, efficiency, and high computational complexity for traditional defect classification methods. This paper focuses on the accuracy and efficiency of classification for steel surface defects. We propose a bag of visual words technique with low computational complexity using BRISK detector and Support Vector Machines. Experiments conducted show that the proposed method outperforms many state-of-the-art approaches.