Diversity of physical encounters and social interactions in urban environments are known to spur economic productivity and innovation in cities, while also to foster social capital and resilience of communities. However, mobility restrictions during the pandemic have forced people to substantially reduce urban physical encounters, raising questions on the social implications of such behavioral changes. In this paper, we study how the income diversity of urban encounters have changed during different periods throughout the pandemic, using a large-scale, privacy-enhanced mobility dataset of more than one million anonymized mobile phone users in four large US cities, collected across three years spanning before and during the pandemic. We find that the diversity of urban encounters have substantially decreased (by 15% to 30%) during the pandemic and has persisted through late 2021, even though aggregated mobility metrics have recovered to pre-pandemic levels. Counterfactual analyses show that while the reduction of outside activities (higher rates of staying at home) was a major factor that contributed to decreased diversity in the early stages of the pandemic, behavioral changes including lower willingness to explore new places and changes in visitation preferences further worsened the long-term diversity of encounters. Our findings suggest that the pandemic could have long-lasting negative effects on urban income diversity, and provide implications for managing the trade-off between the stringency of COVID-19 policies and the diversity of urban encounters as we move beyond the pandemic.