Most childhood cancer survivors do not receive risk-based care through a survivorship program, but factors associated with dropping out of care are unclear. This study aimed to identify characteristics of pediatric cancer survivors who do not return to a cancer center survivorship program for recommended care after at least one visit compared with those who continue to attend. Patient characteristics (demographics, school functioning, psychiatric history) and treatment characteristics (diagnosis, treatment) were abstracted from medical records for all eligible patients. Unadjusted and multivariable logistic regression analyses examined the associations among patient and treatment characteristics and non-attendance. The charts of 400 eligible patients (children < 18, n = 123, adults, n = 277) were reviewed. Of these, 60.3% of patients had not been seen in clinic within one year of their last recommended follow-up appointment. Adult-aged survivors were less likely to return to clinic than child-aged survivors (p < 0.001). For child survivors, longer time off treatment was associated with non-compliance with follow-up. For adult survivors, current age, non-white race, and longer time off treatment were associated with non-compliance. Additional methods to identify survivors at risk for non-compliance with follow-up and interventions for at-risk survivors are needed to improve survivorship care.