For some time, equine-assisted therapy (EAT), i.e., the use of horse-related activities for therapeutic reasons, has been recognised as a useful approach in the treatment of many mental health issues such as post-traumatic stress disorder (PTSD), depression, and anxiety. However, despite the interest in EAT, few scientific studies have focused on understanding the complex emotional response that horses seem to elicit in human riders and handlers. In this work, the potential use of affect recognition techniques based on physiological signals is examined for the task of assessing the interaction between humans and horses in terms of the emotional response of the humans to this interaction. Electroencephalography (EEG), electrocardiography (ECG), and electromyography (EMG) signals were captured from humans interacting with horses, and machine learning techniques were applied in order to predict the self-reported emotional states of the human subjects in terms of valence and arousal. Supervised classification experiments demonstrated the potential of this approach for affect recognition during human-horse interaction, reaching an F1-score of 78.27% for valence and 65.49% for arousal.