While encountering a visual threat, an animal assesses multiple factors to choose an appropriate defensive strategy. For example, when a rodent detects a looming aerial predator, its behavioral response can be influenced by a specific environmental context, such as the availability of a shelter. Indeed, rodents typically escape from a looming stimulus when a shelter is present; otherwise, they typically freeze. Here we report that context-dependent behavioral responses can be initiated at the earliest stage of the visual system by distinct types of retinal ganglion cells (RGCs), the retina’s output neurons. Using genetically defined cell ablation in mature mice, we discovered that some RGC types were necessary for either escaping (alpha RGCs) or freezing (intrinsically photosensitive RGCs) in response to a looming stimulus but not for both behaviors; whereas other RGC types were not required for either behavior (direction-selective RGCs preferring vertical motion). Altogether, our results suggest that specific RGC types regulate distinct behavioral responses elicited by the same threatening stimulus depending on contextual signals in the environment. These findings emphasize the unique contribution of early visual pathways to evolutionally conserved behavioral reactions.