Research has found strong relationships between access to transport, accessibility of activities, and subjective well-being (SWB), and society is said to be car dependent. Accordingly, this study investigates, in a Swedish context, whether and how withdrawal of a driving license for a private car due to visual field loss (VFL) affects SWB. A web survey was used for statistical comparisons of three respondent groups (n = 436): people with a driving license, people with a driving license and VFL, and people whose driving license was withdrawn due to VFL. The inclusion criterion for all participants was that they should have a diagnosis that could cause VFL. The no-license group had lower overall SWB than did respondents with driving licenses. The no-license group also perceived less access to transport means in order to live a life to be satisfied with than did the other groups. The most used transport means in the no-license group was getting a lift in a car, though this group had a strong desire to drive a car. Few respondents in the license groups wanted to use specific transport means to a greater extent, car driving being the most used transport means. Some inter-group differences were seen regarding how access to activities (measured by frequency of actual trips) affected SWB. This study found a significant negative effect of driving license withdrawal on SWB. However, the results imply that qualitative aspects other than the relationship between the frequencies of trips and activities might also affect SWB, and more research on this subject is needed.