This study compared cross-sectional data from online surveys describing the sleep behavior of infants and caregivers in March 2020 (the school closure period during the early stages of the COVID-19 pandemic; n = 295, 23.8 ± 3.8 months old) and March 2019 (before the pandemic; n = 2,017, 24.2 ± 3.8 months old). In comparing those two points in time, no significant differences were found in wake-up times (2019: 7:19 ± 0:46 am vs. 2020: 7:18 ± 0:47 am, p = 0.289), bedtimes (21:01 ± 0:48 pm vs. 21:04 ± 0:53 pm, p = 0.144), or nocturnal sleep times (593.7 ± 43.9 min vs. 588.1 ± 50.3 min, p = 0.613). Regarding the caregivers, in 2020, wake-up times (2019: 6:46 ± 0:50 am vs. 2020: 6:39 ± 0:50 am, p = 0.017) and bedtimes (22:53 ± 1:17 pm vs. 22:42 ± 1:04 pm, p = 0.016) became significantly earlier compared to 2019. Among infants staying at home, total sleep time and percentage of outdoor play decreased significantly, and media use increased significantly in 2020. Lower levels of exercise and more frequent media viewing may have caused prolonged sleep latency in these children. The percentage of caregivers responding with “negative childcare feelings” was significantly higher in the group with less than three nursery school attendance days. Caregivers and infants staying at home are a high-risk group during the pandemic.