This article describes a faculty‐led, multiyear process of formulating learning objectives and assessing the effectiveness of a foreign language requirement for all College of Arts and Sciences undergraduates at a research university. Three interrelated research questions were addressed: (1) What were the levels and patterns of language courses completed under the language requirement compared to those under the previous curriculum? (2) To what extent was the oral proficiency learning objective being attained? and (3) How did oral proficiency vary by course level and the patterns of courses completed to satisfy the language requirement? The oral proficiency of 614 students was assessed with the Simulated Oral Proficiency Interview and categorized in terms of ACTFL ratings. Study findings indicated that 76% of students met or exceeded the objective of the Intermediate Mid level of oral proficiency and that oral proficiency differed by course level and the pattern of courses completed to satisfy the language requirement. In particular, the impact of completing an advanced‐level course was clear, which in turn had implications for curricular policies and academic advising. It is argued that faculty‐led evaluation of program effectiveness, in which assessment approaches are both summative and formative and findings are routinely used to improve educational practices as well as document student learning, is the necessary context for developing an evidence‐based approach to undergraduate language education.