The economic and health crises related to the COVID-19 pandemic raised considerable concern about child and family diet, especially among small-holder farming households in low- and middle-income countries (LMIC). In rural Nepal, 309 families (including 368 children aged 6–66 months) were enrolled pre-COVID-19 in a prospective study of a nutrition education intervention and family milk consumption. The intervention could not be implemented due to COVID-19; however, child and family diet was assessed in three household surveys (one before and two during the pandemic). Over time, after adjusting for child and household factors, child and family diet quality declined (reduced diet diversity, consumption of milk and animal-source-foods (ASF)). However, in dairy-animal-owning (vs. non-dairy-animal-owning) households, both children and family were more likely to consume milk (aOR respectively 2.88× (p < 0.05), 5.81× (p < 0.001)). Similarly, in households producing >3.5 L/d milk (vs. ≤3.5 L/d), children and family members were more likely to consume milk (respectively 7.45× and 11.88× (both p < 0.001)). Thus, the overall decline in child and family diet quality, especially related to milk consumption, was buffered independently by household ownership of ≥1 dairy animals (cow or buffalo) and by milk production >3.5 L/day. A better understanding of these protective factors might facilitate the development of interventions to promote resilience in future crises.