The thermal conductivity and stability of nanofluids pose challenges for their use as coolants in thermal applications. The present study investigates the heat transfer coefficient (HTC) of an Al12Mg17 nanofluid through the utilization of a novel beam displacement method. The study also examines the nanofluid's stability, particle size distribution (PSD), TEM micrograph, and electrical conductivity. From three distinct categories of surfactants, a particular surfactant (CTAB) was chosen to disperse Al12Mg17 nanoparticles in DI water, and subsequently, a two-step method was employed to generate the nanofluid. Dispersion stability is visually monitored and quantified with a zeta potential test. HTC and PSD are measured using optical setups. To evaluate the results, the HTC obtained from the beam displacement method is compared with that of the KD2 Pro apparatus, and the PSD findings are analyzed through TEM micrographs. The results show that a 0.16 vol.% CTAB is the maximum stability for 0.025 vol.% Al12Mg17 nanofluid properly. The optimum ultrasonication period is 2 h, yielding a peak PSD of 154 nm. Increasing nanoparticle concentration enhances HTC up to 40% compared to the base fluid at 0.05 vol.%. Electrical conductivity increases linearly from 155 to 188 μ$${\rm S}/\mathrm{cm}$$
S
/
cm
with nanoparticle concentration. Optical methods for measuring HTC in nanofluids offer the advantage of early results, prior to bulk motion. Thus, the application of nanofluids in thermal systems necessitates the development of optical techniques to improve accuracy.