This study assesses the level of agreement on medication therapy problem (MTP) identification and classification between primary care, ambulatory care pharmacists within a health-system that recently implemented system-wide pharmacist provision of comprehensive medication management (CMM) services. Twenty standardized case vignettes were created and distributed to pharmacists who reviewed each case and identified and categorized MTPs. Outcomes include the number of MTPs identified, identification (yes/no) of specific MTPs within each case (e.g., need for a statin), and Pharmacy Quality Alliance (PQA) category used when classifying MTPs. The level of agreement on MTP identification/categorization was measured using intraclass correlation coefficient (ICC) and interpreted using the Landis and Koch interpretation scale. “Moderate agreement” was observed for the number of MTPs identified by pharmacists (ICC equal to 0.45; 95% confidence interval [CI]: 0.31 to 0.65). In approximately one-half of opportunities, the pharmacists agreed perfectly on the number of MTPs; in approximately one-third of opportunities, the number of MTPs identified varied by 1; and approximately one-tenth of the time, the number of MTPs varied by 2. In regard to the MTP identification (yes/no) and categorization, percent agreement was ≥73% across all MTPs. The results support the need for further training and education and provide the information necessary to target specific disease states.