A pedestrian’s assertiveness when crossing an intersection measures his or her willingness to cross under given conditions, and this level of assertiveness affects pedestrian crossing behavior and safety. Crossing assertiveness at an unsignalized intersection is a complex psychological decision affected by many features, such as the speeds and trajectories of oncoming vehicles, eye contact, facial expression, and hand gesture communications between pedestrians and drivers. To provide a comprehensive understanding of crossing assertiveness of pedestrians at unsignalized intersections, this study applied a pattern recognition method—association rules mining to uncover the patterns for different levels of crossing assertiveness, including assertive, neutral, and passive, using a unique naturalistic driving dataset. An elaborated feature engineering with the decision tree, gradient-boosting decision tree, and XGBoost with SHAP were utilized to select a distinct feature set as input of the Apriori algorithm to recognize the patterns. The results revealed that the driver’s facial expression, the driver’s initiative and passive yield, and the presence of the “yield-to-pedestrian” traffic sign were highly associated with assertive crossing. Features such as the absence of pedestrians on the crosswalk, the presence of incoming speeding vehicles, and the absence of traffic control signs were strongly related to passive crossing. Meanwhile, the number and position of pedestrians at the crosswalk or near the curbside, the communication between pedestrians and drivers, and who actively seeks eye contact were the three major features to convert crossing from neutral to assertive or passive. The results provided a unique and meaningful understanding of pedestrian crossing assertiveness at unsignalized intersections.