Since the very early times of humanity, long before the advent of spoken language, signs and gestures have been in use for communication. Although sign languages are used to bridge the gap wherever vocal communication is impossible, for example to communicate with deaf and mute people as well as with the machines, or difficult if there is a language barrier between the two speakers, there is no optimal work of recognizing sign language in any language as sign languages are vary from country to country. Like any other country Bangladeshi deaf and mute people use their own sign language but unfortunately there are very few works to recognize the sign language which makes the interaction between general and deaf and mute people more tough. In this paper, we proposed an efficient scheme for hand gesture used for Bangla vowel and consonant character recognition in real time adopted by deaf and mute community in Bangladesh. After taking video via webcam as input, hand region is detected by skin color segmentation followed by converting the selected frame into YCbCr. Afterward some preprocessing steps are applied on the image to acquire the region of interest, Hu moment invariants are used to extract features and later on classification and recognizing the character are done by Support Vector Machine (SVM). To use this proposed method as an interpreter for the sign languages of other races no major modifications are required except that the training set should be enriched with desired sign