There have been significant changes in daily activities and corresponding durations since the outbreak of COVID-19. This study examines how the built environment factors and individual/household characteristics affect activity durations (e.g., shopping, social-related, hiking, and working) under the COVID-19 pandemic and analyzes the heterogeneity between different urbanized areas using the data of a Dutch national travel survey in 2020. A hazard-based duration model (e.g., the Cox proportional hazard model) was used to predict activity durations. Estimation results showed that the activity durations for different social groups varied under different geographical and policy conditions. In particular, women and seniors are more susceptible to the unprecedented pandemic, manifested in significantly shorter durations for work and hiking activities. In addition, couples with one or more children need to shorten their working hours and give more attention to their children due to the closure of nurseries and schools. Furthermore, the influences of built environment factors also present significant differences. A higher number of service facilities does not significantly foster the extension of hiking activity duration; however, this is the opposite among regions with more open green areas. Compared with previous studies on analyzing the influencing factors of activity durations, this study incorporated some unique variables (e.g., COVID-19 countermeasures and urban class) to consider the temporal and spatial heterogeneity under the particular pandemic period.