School bullying and cyberbullying victimization and perpetration are prevalent in adolescents with autism spectrum disorder (AASD). However, the levels of adolescent–caregiver agreement regarding the bullying involvement of AASD and the factors associated with these levels remain to be evaluated. In the present study, we evaluated the levels of adolescent–caregiver agreement on the school bullying and cyberbullying involvement experiences of AASD and the factors associated with the levels of agreement. This study included 219 dyads of AASD and their caregivers. The school bullying and cyberbullying involvement experiences of the participating AASD were assessed using the School Bullying Experience Questionnaire and the Cyberbullying Experiences Questionnaire, respectively. Attention-deficit/hyperactivity disorder, oppositional defiant disorder (ODD), depressive and anxiety symptoms, and autistic social impairment were also assessed. AASD and their caregivers had poor to fair levels of agreement regarding the school bullying and cyberbullying victimization and perpetration experiences of AASD. Severe inattention, hyperactivity–impulsivity, ODD, depressive and anxiety symptoms, and autistic social impairment were associated with high levels of adolescent–caregiver agreement. When assessing the bullying involvement experiences of AASD, mental health professionals should obtain information from multiple sources. In addition, the factors influencing the levels of agreement should be considered.