Cyclist noise exposure has implications for health, comfort, and safety. Methods used for in situ measurement of on-road noise levels for cyclists vary, and the effects of key study design factors have not been investigated. To enable reliable research into cyclist noise exposure, this study aims to determine the accuracy of smartphone noise measurements in comparison with a sound level meter (SLM) reference instrument, and how noise levels are affected by travel speed, air speed, sensor placement, and use of a windscreen. Field data were collected with paired instruments in a typical urban cycling scenario, and comparisons made varying one design factor at a time (smartphone versus SLM, with versus without windscreen, handlebar versus shoulder placement, etc.). Results show that smartphones can generate reliable measurements (compared with SLM) of high-resolution (1-s) cyclist exposure for C-weighted noise, but not A-weighted noise. Sensor placement and windscreen have small effects on noise readings, but air speed and travel speed greatly affect measured noise levels. Future studies measuring on-road noise must consider the effects of wind- and bicycle-generated noise to ensure internal validity. Studies should also consider both study objectives and instrumentation when selecting a noise measure (frequency weighting). Research is needed into bicycle noise generation and perception of traffic noise by cyclists to enhance the reliability of future studies.