Nitrogen oxide (NOx) emissions from heavy-duty diesel vehicles (HDDVs) have adverse effects on human health and the environment. On-board monitoring (OBM), which can continuously collect vehicle performance and NOx emissions throughout the operation lifespan, is recognized as the core technology for future vehicle in-use compliance, but its large-scale application has not been reported. Here, we utilized OBM data from 22,520 HDDVs in China to evaluate their real-world NOx emissions. Our findings showed that China VI HDDVs had a 73% NOx emission reduction compared with China V vehicles, but a considerable proportion still faced a significant risk of higher NOx emissions than the corresponding limits. The unsatisfactory efficiency of the emission treatment system under disadvantageous driving conditions (e.g., low speed or ambient temperature) resulted in the incompliance of NOx emissions, especially for utility vehicles (sanitation/garbage trucks). Furthermore, the observed intertrip and seasonal variability of NOx emissions demonstrated the need for a long-term continuous monitoring protocol instead of instantaneous evaluation for the OBM. With both functions of emission monitoring and malfunction diagnostics, OBM has the potential to accurately verify the in-use compliance status of largescale HDDVs and discern the responsibility of high-emitting activities from manufacturers, vehicle operators, and driving conditions.