Automated driving in urban environments not only has the potential to improve traffic flow and heighten driver comfort but also to increase traffic safety, particularly for vulnerable road users such as pedestrians. For these benefits to take effect, drivers need to trust and use automated vehicles. This decision is influenced by both system and context factors. However, it is not yet clear how these factors interact with each other, especially for automated driving in city scenarios with crossing pedestrians. Therefore, we conducted an online experiment in which participants (N = 68) experienced short automated rides from the driver’s perspective through an urban environment. In each of the presented videos, a pedestrian crossed the street in front of the automated vehicle while system and context factors were varied: 1) the crossing pedestrian’s intention was either visualized correctly (as crossing) or incorrectly (visualization missing) by the automated vehicle (system factor), 2) the pedestrian was either distracted by using a smartphone while crossing or not (context factor), and 3) the scenario was either more or less complex depending on the number of other vehicles and pedestrians being present (context factor). In situations with a system malfunction where the crossing pedestrian’s intention was not visualized, participants perceived the situation as more critical, had less trust in the automated system, and a higher willingness to take over control regardless of any context factors. However, when the system worked correctly, the crossing pedestrian’s smartphone usage came into play, especially in the less complex scenario. Participants perceived situations with a distracted pedestrian as more critical, trusted the system less, indicated a higher willingness to take over control, and were more uncertain about their decision. As this study demonstrates the influence of distracted pedestrians, more research is needed on context factors and their inclusion in the design of interfaces to keep drivers informed during automated driving in urban environments.