The necessary inspection of railway tracks currently still requires high effort and costs, even though cheap sensors and IoT-capable devices are widely available. Such devices could be installed in regular in-service rail vehicles and potentially monitor and measure track irregularities comparably well. This would enable new possibilities for railway operators as they could provide services that are traditionally executed by network operators. Smartphones are a typical example of these kind of devices. Existing studies already researched upon the usage of smartphones in rail vehicles to potentially measure track irregularities. However, from our knowledge none of the previous studies assessed the quality of smartphone accelerometers. Therefore, we conducted experiments on a shaker test rig with different smartphones to study the frequency response of different devices. This paper presents the results of these experiments and shows that smartphones are in general not suitable to measure track irregularities directly. We show that the quality of the data only allows for monitoring applications with the focus of detecting larger deviations over time. The exact calculation of deviations respectively track irregularities is not viable from our perspective.