The COVID-19 pandemic has brought unprecedented changes to our mobility. It has not only changed our work-related travel patterns but also impacted leisure and other utilitarian activities. Non–work-related trips tend to be more seriously affected by the neighborhood/contextual factors such as socioeconomic status (SES), and destination accessibility, and COVID-19 impact on non-work trips may not be equal across different neighborhood SES. This study compares non–work-related travel patterns between the pre- and during COVID-19 pandemic in the City of El Paso, Texas. By utilizing smartphone mobility data, we captured the visitation data for major non-work destinations such as restaurants, supermarkets, drinking places, religious organizations, and parks. We used Census block groups ( n = 424) within the city and divided them into low- and high-income neighborhoods based on the citywide median. Overall, the total frequency of visitations and the distances traveled to visit these non-work destinations were influenced by the COVID-19 pandemic. However, significant variations existed in their visitation patterns by the type of non-work destinations. While the overall COVID-19 effects on non-work activities were evident, its effects on the travel patterns to each destination were not equal by neighborhood SES. We also found that COVID-19 had differently influenced non-work activities between high- and low-income block groups. Our findings suggest that the COVID-19 pandemic may exacerbate neighborhood-level inequalities in non-work trips. Thus, safe and affordable transportation options together with compact and walkable community development appear imperative to support daily travel needs for various utilitarian and leisure purposes, especially in low-income neighborhoods.