Automatic pain recognition is essential in healthcare. In previous studies, automatic pain recognition methods preferentially apply the features extracted from physiological signals for conventional models. This method provides good performance but mainly relies on medical expertise for feature extraction of physiological signals. This paper presents a deep learning approach based on physiological signals that have the role of both feature extraction and classification, regardless of medical expertise. We propose multi-level context information for each physiological signal discriminating between pain and painlessness. Our experimental results prove that multi-level context information has more significant performances than uni-level context information based on Part A of the BioVid Heat Pain database and the Emopain 2021 dataset. For Part A of the BioVid Heat Pain database, our experimental results for pain recognition tasks include: Pain 0 and Pain 1, Pain 0 and Pain 2, Pain 0 and Pain 3, and Pain 0 and Pain 4. In the classification task between Pain 0 and Pain 4, the results achieve an average accuracy of 84.8 ± 13.3% for 87 subjects and 87.8 ± 11.4% for 67 subjects in a Leave-One-Subject-Out cross-validation evaluation. The proposed method adopts the ability of deep learning to outperform conventional methods on physiological signals. INDEX TERMS pain recognition, physiological signals, context vector, attention module, deep learning.