In the recent decades, traditional concrete poses a great challenge to the modernization of the construction industry because of low tensile strength, poor toughness, and weak resistance to cracking. To overcome these problems, ultra-high performance concrete (UHPC) with superior mechanical properties and durability is developed for broad application prospect in the future engineering construction. However, UHPC is less eco-friendly because it consumes more cement compared with the traditional concrete. The manufacturing of cement produces large amounts of carbon dioxide and therefore leads to the greenhouse effect. Nanomaterials consist of microstructural features that range from 0.1 to 100 nm in size, which exhibit the novel properties different from their bulk counterparts, including filling effect, surface activity, and environmental sustainability. This paper reviews the effect of various nanomaterials used in UHPC to partially replace the cement or as an additive on the microstructures, mechanical properties, and other properties of UHPC. In addition, the limitations and shortcomings of the current research are analyzed and summarized, and development directions are provided for future research on the application of nanomaterials in UHPC.