The COVID-19 pandemic has affected us in numerous ways and may consequently impact our relationships with pet dogs and cats. We conducted a longitudinal survey to examine the temporal patterns of owner-pet relationship, stress, and loneliness during four phases of the pandemic: 1) pre-pandemic (February 2020), 2) lockdown (April to June 2020), 3) reopening (September to December 2020), and 4) recovery (January 2021 to December 2021). We also investigated the effect of pet ownership on stress and loneliness, by considering a set of a priori causal assumptions. In addition, we hypothesized that the differences in the levels of stress and loneliness between dog and cat ownerships were mediated by the owner-pet relationship. A total of 4,237 participants (657 non-pet owners, 1,761 dog owners, and 1,819 cat owners) completed between one and six surveys. Overall, the closeness in the relationship between owners and their pets increased with time during the study period. We also observed that dog owners consistently showed larger decreases in the levels of stress and loneliness than cat and non-pet owners. However, after adjusting for confounders, the findings did not support a mitigating effect of pet ownership. Pet ownership did not alleviate stress, social loneliness resulting from a lack of friendships or workplace relationships, or emotional loneliness due to deficiencies in family relationships. Pet owners, however, reported a lower degree of emotional loneliness caused by deficits in romantic relationships than non-pet owners. Our results also indicated that the differences in stress and loneliness levels between dog and cat ownerships were partially explained by the owner-pet relationship, and once this was accounted for, the differences between them reduced. In summary, this study highlights the dynamic effects of COVID-19 on owner-pet relationship and mental health. It also shows the complexity of the association between pet ownership and mental health, partially mediated by owner-pet relationships.