Recent research has shown promising results for the detection of aortic stenosis (AS) using cardio-mechanical signals. However, they are limited by two main factors: lacking physical explanations for decision-making on the existence of AS, and the need for auxiliary signals. The main goal of this paper is to address these shortcomings through a wearable inertial measurement unit (IMU), where the physical causes of AS are determined from IMU readings. To this end, we develop a framework based on seismo-cardiogram (SCG) and gyro-cardiogram (GCG) morphologies, where highly-optimized algorithms are designed to extract features deemed potentially relevant to AS. Extracted features are then analyzed through machine learning techniques for AS diagnosis. It is demonstrated that AS could be detected with 95.49–100.00% confidence. Based on the ablation study on the feature space, the GCG time-domain feature space holds higher consistency, i.e., 95.19–100.00%, with the presence of AS than HRV parameters with a low contribution of 66.00–80.00%. Furthermore, the robustness of the proposed method is evaluated by conducting analyses on the classification of the AS severity level. These analyses are resulted in a high confidence of 92.29%, demonstrating the reliability of the proposed framework. Additionally, game theory-based approaches are employed to rank the top features, among which GCG time-domain features are found to be highly consistent with both the occurrence and severity level of AS. The proposed framework contributes to reliable, low-cost wearable cardiac monitoring due to accurate performance and usage of solitary inertial sensors.