As a novel mode of urban air mobility (UAM), unmanned aerial vehicles (UAVs) pose a great amount of risk to ground people. Assessing ground risk and mitigation effects correctly is a focused issue. This paper proposes a grid-based risk matrix framework for assessing the ground risk associated with two types of UAVs, namely fixed-wing and quadrotor. The framework has a three-stage structure of “intrinsic risk assessment—mitigation effect—final map generation”. First, the intrinsic risk to ground populations caused by potential UAV crashes is quantified. Second, the mitigation effects are measured by establishing a mathematical model with a focus on the ground sheltering and parachute systems. Finally, a modular approach is presented for generating a ground risk map of UAVs, aiming to effectively characterize the effects of each influencing factor on the failure process of UAVs. The framework facilitates the modular analysis and quantification of the impact of diverse risk factors on UAV ground risk. It also provides a new perspective for analyzing ground risk mitigation measures, such as ground sheltering and UAV parachute systems. A case study experiment on a realistic urban environment in Shenzhen shows that the risk map generated by the presented framework can accurately characterize the distribution of ground risk posed by various UAVs.