Phase transformations of Al doped NbN nanocomposite coatings are studied in detail focusing on their microstructural evolution and phase composition. Several techniques such as XRD, SEM, HR-TEM, Nanoindentation and molecular dynamics simulation are employed in order to understand the phase evolution of the Nb-Al-N system. The nanocomposite structures were formed in the coatings, the roughness of the coatings decreased with increasing the Al concentration due to decreasing grain size. First-principles investigation of Nb-Al-N solid solutions was carried out to interpret film properties. It was found, that for small Al fractions, the solid solutions will form in agreement with our experimental results. The spinodal decomposition of Nb-Al-N solid solutions is supposed to be responsible for the formation of the nanocomosite structure observed in the deposited Nb-Al-N films.