An Automatic Speech Recognition System is developed for addressing the Tamil conversational speech data of the elderly people and transgender. The speech corpus used in this system is collected from the people who adhere their communication in Tamil at some primary places like bank, hospital, vegetable markets. Our ASR system is designed with pre-trained model which is used to recognize the speech data. WER(Word Error Rate) calculation is used to analyse the performance of the ASR system. This evaluation could help to make a comparison of utterances between the elderly people and others. Similarly, the comparison between the transgender and other people is also done. Our proposed ASR system achieves the word error rate as 39.65%.