Ultra-fine filler or mineral powder is the main mineral component responsible for structure formation in the bitumen-mineral system, therefore mineral and chemical composition, chemical reactivity, surface area, fineness, particle shape, porosity and density are the crucial parameters for structure formation of the composite. This work studied the effect of fineness and chemical and mineral composition of fillers on the structure of asphalt binder. It was demonstrated that an increase in surface area boosts porosity, and void content of the filler, but reduces the porous size. For carbonate fillers such as limestone and chalk with high fineness it was investigated that compaction applied to asphalt binder specimens showed very low water saturation. This can be explained by the film effect of water impermeable bitumen in the matrix and by small porous size with mostly close pores. An increase in surface area of silicate fillers improves the compaction of structure but cannot reach the same level of compaction degree demonstrated by the specimens with carbonate fillers. SEM analysis of microstructural characteristics for the asphalt binder showed that the incorporation of fine-fractioned chalk filler resulted in the formation of asphalt binder with high density and micro-and nanoporous matrix.