Having recently become an important research topic, indoor tracking in a multi-floor building delivers comprehensive and efficient location-based services. In this paper, we present a deep learning (DL)based indoor multi-floor tracking scheme, that is independent of infrastructure and only uses the smartphone as a terminal device to measure and analyze the user's mobility information. Our method detects the floor transition according to changes in barometer readings. We compiled the time-series barometer data to train the DL model, and applied the data augmentation method to avoid overfitting and data imbalances during model training. Furthermore, we developed a floor decision algorithm to process the DL model's output and generate the floor detection result. In the proposed scheme, the smartphone's inertial measurement unit sensors are used to measure the user's mobility information, and pedestrian dead reckoning (PDR) is exploited to update the user's 2D location. We implemented the multi-floor tracking by combining the floor detection algorithm with PDR. To avoid the accumulated error problem that commonly arises in the infrastructure-free approach, the calibration nodes (CN) were configured in the floor plan to correct the estimated location by matching the possible CNs during the floor transition. We conducted several experiments in multi-floor buildings to evaluate our scheme's performance, and found that our floor detection method achieves a 99.6% average floor number accuracy, with all floor transition types (i.e., stairs, elevator) being successfully recognized. Furthermore, we compared localization performance with the conventional methods to validate the effectiveness of our approach.