The influence of Zn and Si atoms on AlGaInAs compositionally graded buffers (CGBs) was studied using the approaches of optical microscopy, reciprocal space mapping, Raman spectroscopy and cathodoluminescence imaging. CGBs with Zn or Si doping could achieve good relaxation of about 90%, while Si but not Zn doping significantly impacted the tilt angle. Ribbon-like scratches along the 〈1–10〉 direction were observed in samples with Si doping. Sharp increases in threading dislocation density (TDD) and dislocation pile-up were observed along the scratches. In contrast, CGBs with Zn doping featured smooth surfaces and lower TDDs, contributing to improved chip performance. Based on these results, we conclude that Si atoms decrease the glide length of misfit dislocation segments and cause dislocation nucleation multiplication and dislocation pile-up and eventually increase surface roughness and TDD, while Zn could serve as a surface surfactant medium.