More and more consumer devices implement the IEEE Ultra-Wide Band (UWB) standard to perform distance measurements for sensitive tasks such as keyless entry and startup of modern cars, to find lost items using coin-sized trackers, and for smart payments. While UWB promises the ability to perform time-of-flight centimeter-accurate distance measurements between two devices, the accuracy and reliability of the implementation in up-to-date consumer devices have not been evaluated so far. In this paper, we present the first evaluation of UWB smartphones from Apple, Google, and Samsung, focusing on accuracy and reliability in passive keyless entry and smart home automation scenarios. To perform the measurements for our analysis, we build a custom-designed testbed based on a Gimbal-based platform for Wireless Evaluation (GWEn), which allows us to create reproducible measurements. All our results, including all measurement data and a manual to reconstruct a GWEn are published online. We find that the evaluated devices can measure the distance with an error of less than 20 cm, but fail in producing reliable measurements in all scenarios. Finally, we give recommendations on how to handle measurement results when implementing a passive keyless entry system.• Reliability and Accuracy of UWB technology. People using UWB as a PKE system need to know if they can rely on accurate measurements. Without this, people will be less likely to adopt the technology and more likely to resist its implementation. A positive evaluation will provide peace of mind to people using a UWBsmartphone as a car key. • Implementation of UWB measurements. Manufacturers