Filler’s presence in hot-mix asphalt (HMA) is even though minimal but they do affect its durability characteristics. Many natural and waste materials in the form of fillers have been studied for their effectiveness on HMA mix design and performance characteristics. However, in practice, stone dust (SD) is the preferred filler due to its abundance, ease of availability, and cost-effectiveness. Thus, the major objective of this study was to investigate the effect of locally available materials: stone dust (SD)- natural, hydrated lime (HL) – processed, rice-husk ash (RHA) and fly-ash (FHA)-waste materials on HMA properties based on the factors such as availability, field utilization, cost, and sustainability, while at the same time identify the anomalies of those selected fillers on HMA mix if any. A viscosity grade (VG-30) binder was selected and checked for its fundamental consistency characteristics set forth in Indian standards. In this study, aggregate gradation structure specified as bituminous concrete grading 1 (BC1) in India was designed for the preparation and evaluation of four HMA mixes: (a) BC1 with SD (BC1-SD), (b) BC1 with RHA (BC1-RHA), (c) BC1 with FA (BC1-FA), and d) BC1 with HL (BC1-HL). Fillers: RHA, FA, and HL were studied for their physico-chemical properties. The most recommended filler dosage of 4% by weight of mix was selected and kept uniform for the various BC1 mixes. Marshall method of mix design was performed to identify the optimum asphalt content (OAC) of four different BC1 mixes. The test results of methylene blue value (MBV), german filler value (GFV), and fineness modulus (FM) indicated that RHA includes more micron-to-nano sized particles than the other two fillers (HL and FA). The scanning electron microscope coupled with energy dispersive X-ray results showed that the RHA and FA exhibited similar chemical composition, while HL was identified to be a calcium-based compound. The BC1-RHA mix resulted in non-cohesive mix for the binder content ranging from 4.5 to 6.5%. Additionally, for the binder contents in the range of 7 to 9% the BC1-RHA compacted samples failed to yield air voids of 4% required to arrive at the OAC. The BC1- FA mix showed the highest Marshall stability (26.97 KN) followed by BC1-HL (23.97 kN), and BC1-SD (17.9 kN). Also, retained stability test results of all the three different mixes were in close proximity to each other indicative of the affinity of the fillers to asphalt. The resistance to moisture susceptibility results indicated that HL is the better anti-stripping element followed by FA, and SD. Among the three different filler-based BC1 mixes, BC1-HL mix was adjudged as an effective moisture resistant mix followed by BC1-FA, and BC1-SD. However, a single filler that not only tends to improve the various performance parameters of the mix but be available in abundance and cost-effective is yet to be explored.