Background: Excessive time spent playing video games is associated with adverse health outcomes in adolescents. Although poor child–parent relationship and social relations with peers are considered as possible predictors, little is known as to whether adverse childhood experiences (ACEs) are associated with time spent playing video games. The aim is to examine the association between ACEs and time spent playing video games in adolescents. Methods: We used pooled data from the Adachi Child Health Impact of Living Difficulty (A-CHILD) study in 2016 and 2018, which is a population-based cross-sectional study in Adachi City, Tokyo, Japan (N = 6799, 4th, 6th, and 8th-grade students). Adolescents answered questionnaires examining the time spent playing video games, per day, on weekdays (“less than 1 h”, “less than 3 h”, and “more than 3 h”) and ACEs (eight types). Results: The results of the ordinal logistic regression analysis showed a positive association between ACE total score and time spent playing video games after adjusting for covariates (1 ACE: OR = 1.28, 95% CI = 1.10–1.48; 2 ACEs: OR = 1.25, 95% CI = 1.06–1.48; 3 + ACEs: OR = 1.44, 95% CI = 1.14–1.82, p for trend < 0.001). Regarding each type of ACE, the experiences of single parenthood, parental history of psychiatric disorders, and peer isolation were independently positively associated with time spent playing video games. Conclusions: Health policy to address ACEs might be important to shorten the time spent playing video games.