Studies of long-term trends in phenology often rely on climatic averages, overlooking climate variability. Here we test the hypothesis that unusual weather conditions are particularly critical in driving adult insect phenology. First, we generated phenological estimates for Lepidoptera (moths and butterflies) across the Eastern USA, and over a 70 year period, using natural history collections data. Next, we assembled a set of key predictors, including the number of unusually warm and cold days prior to, and during, adult flight period. We then used phylogenetically informed linear mixed effects models to evaluate effects of unusual weather events, climate context, species traits, and their interactions on the flight onset, offset and duration. We found increasing numbers of both warm and cold days was the strongest climate driver in our models, dramatically increasing flight duration. This strong effect is driven by differential onset and termination dynamics. For flight onset, impact of unusual climate conditions is dependent on climatic context, but for flight cessation, more unusually cold days always lead to later termination. These results show that understanding phenological responses under global change must account for unusual or anomalous weather events, especially given these events are predicted to increase in frequency and severity.