Many people have worried about COVID-19 infection, job loss, income reduction, and family conflict during the COVID-19 pandemic. Some social groups may be particularly vulnerable due to their residential neighborhoods and daily activities. On the other hand, people’s daily exposure to greenspace offers promising pathways for reducing these worries associated with COVID-19. Using data collected with a questionnaire and a two-day activity diary from two typical neighborhoods in Hong Kong, this study examines how people’s housing conditions and daily greenspace exposure affect their perceived COVID-19 risk and distress (i.e., worries about job loss, income reduction, and family conflict) during the pandemic. First, the study compares people’s perceived COVID-19 risk and distress based on their residential neighborhoods. Further, it examines the associations between people’s perceived COVID-19 risk and distress with their housing conditions and daily greenspace exposure using ordinal logistic regression models. The results indicate that living in a high-risk neighborhood, being married, renting a residential unit, and living in a large household are significantly associated with a higher neighborhood-based perceived COVID-19 risk and distress during the pandemic. In addition, people also reported lower mobility-based perceived COVID-19 risk when compared to their neighborhood-based perceived COVID-19 risk, while they still have a high perceived COVID-19 risk in their occupational venues if they have to work in a high-risk district (e.g., Kowloon). Lastly, daily greenspace exposure (i.e., woodland) could reduce people’s perceived COVID-19 risk and distress. These results have important implications for the public health authority when formulating the measures during the COVID-19 pandemic.