The dependences of the surface roughness and the phase structure of compound thin films on substrate temperature and flux of incoming particles are investigated by a proposed mathematical model. The model, which describes physically deposited thin compound film growth process is based on the Cahn–Hilliard equation and includes processes of phase separation, adsorption, and diffusion. In order to analyze large temperature range and assuming deposition of energetic particles, the diffusion is discriminated into thermal diffusion, radiation-enhanced diffusion, and ion beam mixing. The model is adapted to analyze surface roughness evolution during film growth. The influences of the substrate temperature and incoming flux particles on the surface roughness are determined by a series of numerical experiments. The modelling results showed that the surface roughness increased as the substrate temperature rose. Besides, a similar relationship was discovered between substrate temperature and size of nanoparticles formed in binary films, so the increase in the surface roughness with the substrate temperature was attributed to the increase in size of nanoparticles.