Cement is one of the main constituents of concrete material and it is one of the main sources of carbon dioxide emissions in the environment. Fillers within a range of 5–7% from different sources can be used as a replacement of cement without compromising the properties of concrete or even tailoring for required property. This paper investigates the influence of inactive silica filler and silica fume on the mechanical- and durability-related properties of concrete with different strengths. The investigated mechanical properties focused on compressive strength at different ages up to 400 days, while the durability-related properties focused on porosity and rapid chloride ion penetrability (RCPT). Two types of ultrafines, namely quartz ultrafine and silica fume, were used. Concrete mixtures with four different water/binder ratios (0.25, 0.30, 0.35, 0.40) were prepared for various dosages of quartz ultrafine (0%, 5%, 8%, 10%, 15%, 25%, and 35%) and different dosages of silica fume (0%, 8%, 10%, and 12%). The results revealed that the compressive strength and durability related properties of concrete with different dosages of ultrafines and silica fume were significantly affected; however, there was a negative impact of ultrafine filler on the compressive strength after replacement of more than 8% of ultrafines. The strength relationships for the concrete with different water-to-cement ratio were assessed and certain modifications were proposed for ultrafines and silica fume. Predictive models were proposed for predicting the compressive strength of concrete in terms of RCPT and porosity for different levels of replacements of ultrafines and silica fume.