Evaluation of outcome measures can provide policymakers with valuable information on the effectiveness of psychiatric rehabilitation. Two specific challenges in collecting outcome measures for psychiatric rehabilitation programs are heterogeneity of outcomes and difficulty with follow-up. These two challenges were illustrated in the process of evaluating Rose Hill Center, a rural residential psychiatric rehabilitation program. The original design was to conduct interviews with former residents and family members and verify healthcare utilization. The difficulty of locating people and their reluctance to participate conspired to lower the follow-up rates. The design was modified to improve the follow-up rate but decrease the details of specific outcomes. The results showed a high but biased follow-up rate, with more information obtained for people who graduated from the program. The residents with planned discharge showed excellent outcomes in terms of living situation, working situation, healthcare utilization, and low severity of current symptoms. High compliance with medication spoke to the program's philosophy of including the residents in the decision-making process. The dual challenges of heterogeneity of outcomes and difficulty in follow-up that limit efforts to document the value of psychiatric rehabilitation are discussed.