The rapid development of smartphone sensors has provided rich indoor pedestrian trajectory data for indoor location-based applications. To improve the quality of these collected trajectory data, map matching methods are widely used to correct trajectories. However, these existing matching methods usually cannot achieve satisfactory accuracy and efficiency and have difficulty in exploiting the rich information contained in the obtained trajectory data. In this study, we proposed a novel semantic matching method for indoor pedestrian trajectory tracking. Similar to our previous work, pedestrian dead reckoning (PDR) and human activity recognition (HAR) are used to obtain the raw user trajectory data and the corresponding semantic information involved in the trajectory, respectively. To improve the accuracy and efficiency for user trajectory tracking, a semantic-rich indoor link-node model is then constructed based on the input floor plan, in which navigation-related semantics are extracted and formalized for the following trajectory matching. PDR and HAR are further utilized to segment the trajectory and infer the semantics (e.g., "Turn left", "Turn right", and "Go straight"). Finally, the inferred semantic information is matched with the semantic-rich indoor link-node model to derive the correct user trajectory. To accelerate the matching process, the semantics inferred from the trajectory are also assigned weights according to their relative importance. The experiments confirm that the proposed method achieves accurate trajectory tracking results while guaranteeing a high matching efficiency. In addition, the resulting semantic information has great application potential in further indoor location-based services.