The use of smartphone tracking is seen as the way forward in data collection for travel behavior studies. It overcomes some of the weaknesses of the classical approach (which uses paper trip diaries) in terms of accuracy and user annoyance. This article evaluates if these benefits hold in the practical application of smartphone tracking and compares the findings of a travel behavior survey using smartphone tracking to the findings of a previous paper survey. We compare three phases of the travel behavior study. In the recruitment phase, we expect smartphone tracking to make people more willing to participate in surveys, given the innovative nature and reduced burden to participants. However, we found the recruitment of participants equally challenging as for classical methods. In the data collection phase, however, we observe that participants entering the smartphone tracking survey are much more likely to complete the data collection period than when using paper trip diaries. Because of the limited burden, the risk of drop-out from the survey is significantly lower, making the actual data collection more efficient, even for longer survey periods. Finally, in the data analysis phase, the travel behavior indicators derived from smartphone tracking data result in higher average trip rates, shorter average trip lengths and a higher share of active modes (bike, walking) than the results from the paper survey. Although this is explained by more complete and more consistent trip registration, this finding is problematic for comparability between surveys based on different methods, both for longitudinal monitoring (comparability over consequent surveys) and for benchmarking (comparability over geographical areas). Therefore, it is crucial to clearly report the applied data collection methods when describing or comparing travel indicators. In surveys, a combined approach of both written trip diaries and smartphone tracking is advised, where each method can complement the shortcomings of the other.