In addition to the cognitive, developmental, and pedagogical challenges of learning a second language, foreign language learners also face emotional obstacles. Negative emotions such as confusion, frustration, and anxiety interfere with the cognitive channels, which in return, moderates the learning. These emotions also diminish the sense of self-efficacy, producing a disinclination toward beneficial practice. In this research, we are focusing on foreign language anxiety (FLA). Our long-term goal is to recognize FLA and overcome it within the context of an intelligent foreign language learning system. As a first step, we are focusing on the recognition of FLA within a simple e-learning system for English language learners. To identify FLA, we correlated physical measurements (eye fixation, heart rate, and blood pressure) with emotional self-report. We found that the compound predictors of self-report of language difficulty, self-report of system difficulty, and exercise score can identify language anxiety. This finding will allow us to build a sensor-free anxiety detector which we will use to help overcome foreign language anxiety in the future.