Earthquakes pose substantial threats to communities worldwide. Understanding how people respond to the fast-changing environment during earthquakes is crucial for reducing risks and saving lives. This study aims to study people’s protective action decision-making in earthquakes by leveraging explainable machine learning and video data. Specifically, this study first collected real-world CCTV footage and video postings from social media platforms, and then identified and annotated changes in the environment and people’s behavioral responses during the M7.1 2018 Anchorage earthquake. By using the fully annotated video data, we applied XGBoost, a widely-used machine learning method, to model and forecast people’s protective actions (e.g., drop and cover, hold on, and evacuate) during the earthquake. Then, explainable machine learning techniques were used to reveal the complex, nonlinear relationships between different factors and people’s choices of protective actions. Modeling results confirm that social and environmental cues played critical roles in affecting the probability of different protective actions. Certain factors, such as the earthquake shaking intensity and number of people shown in the environment, displayed evident nonlinear relationships with the probability of choosing to evacuate. These findings can help emergency managers and policymakers design more effective protective action recommendations during earthquakes.