Recently, the machine learning technologies have been widely used in cognitive applications such as Automatic Speech Recognition (ASR) and Image Recognition (IR). Unfortunately, these techniques have been massively used in unauthorized audio/image data analysis, causing serious privacy leakage. To address this issue, we propose HAMPER in this work, which is a data encryption framework that protects the audio/image data from unauthorized ASR/IR analysis. Leveraging machine learning models' vulnerability to adversarial examples, HAMPER encrypt the audio/image data with adversarial noises to perturb the recognition results of ASR/IR systems. To deploy the proposed framework in extensive platforms (e.g. mobile devices), HAMPER also take into consideration of computation efficiency, perturbation transferability, as well as data attribute configuration. Therefore, rather than focusing on the high-level machine learning models, HAMPER generates adversarial examples from the low-level features. Taking advantage of the light computation load, fundamental impact, and direct configurability of the low-level features, the generated adversarial examples can efficiently and effectively affect the whole ASR/IR systems. Experiment results show that, HAMPER can effectively perturb the unauthorized ASR/IR analysis with 85% Word-Error-Rate (WER) and 83% Image-Error-Rate (IER) respectively. Also, HAMPER achieves faster processing speed with 1.5X speedup for image encryption and even 26X in audio, comparing to the state-of-the-art methods. Moreover, HAMPER achieves strong transferability and configures adversarial examples with desired attributes for better scenario adaptation.