Emotion recognition from speech is experiencing different research applications. It is becoming one of the tool for analysis of health condition of the speaker. In this work, the emotions such as anger, fear, happy, neutral are considered for speech emotion algorithm design. A database built by IITKGP is used for emotion recognition. For any recognition, feature extraction and pattern classification are the important tasks. In this work the features considered are Mel Frequency Cepstral Coefficients (MFCC), Pitch chroma, prosodic are used. Hidden Markov Models (HMMs) are used to for modeling and identify the emotions. In this research work, the database considered for emotion recognition is taken in different combinations such as male training-female testing, male training-male testing, female training-female testing, female training-male testing. All these combinations are trained and tested with i-vector with GMM, linear Hidden Markov Models (HMMs) and Ergodic Hidden Markov Models(EHMMs) In almost all the cases, Ergodic Hidden Markov Models (EHMMs) method has shown significant improvement in recognition accuracy than i-vector with GMM and Linear Hidden Markov Models(HMMs)