Emotion recognition is one of the biggest challenges in human-human and human-computer interaction. There are various approaches to recognize emotions like facial expression, audio signals, body poses, and gestures etc. Physiological signals play vital role in emotion recognition as they are not controllable and are of immediate response type. In this paper, we discuss the research done on emotion recognition using skin conductance, skin temperature, electrocardiogram (ECG), electromyography (EMG), and electroencephalogram (EEG) signals. Altogether, the same methodology has been adopted for emotion recognition techniques based upon various physiological signals. After survey, it is understood that none of these methods are fully efficient standalone but the efficiency can be improved by using combination of physiological signals. The study of this paper provides an insight on the current state of research and challenges faced during emotion recognition using physiological signals, so that research can be advanced for better recognition.