We address the indoor tracking problem by combining an Impulse Radio-Ultra-Wideband handset with an ankle-mounted Inertial Measurement Unit embedding an accelerometer and a gyroscope. The latter unit makes possible the detection of the stance phases to overcome velocity drifts. Regarding radiolocation, a time-of-arrival estimator adapted to energy-based receivers is applied to mitigate the effects of dense multipath profiles. A novel quality factor associated with this estimator is also provided as a function of the received signal-to-noise ratio, enabling us to identify outliers corresponding to obstructed radio links and to scale the covariance matrix of radiolocation measurements. Finally, both radio and inertial subsystems are loosely-coupled into one single navigation solution relying on a specific extended Kalman filter. In the proposed fusion strategy, processed inertial data control the filter state prediction whereas Combined Time Differences Of Arrival are formed as input observations. These combinations offer low computational complexity as well as a unique filter structure over time, even after removing outliers. Experimental results obtained in a representatively harsh indoor environment emphasize the complementarity of the two technologies and the relevance of the chosen fusion method while operating with low-cost, noncollocated, asynchronous, and heterogeneous sensors.