This work was conducted to investigate lattice, band, and spin engineering of magnetic Zn 1Àx Co x O layers towards quantum barriers in ZnO. Lattice distortions by doping with Co ions caused a flat tetrahedron in the host, leading to an increase of spontaneous polarization in Zn 1Àx Co x O compared to ZnO based on the point-charge model. The band-gap energy increased linearly with the Co concentrations, which was very similar to the band-gap widening in Zn 1Àx Mg x O derived from sp hybridization. The Co (3d) states were located in the mid-gap, which remained unchanged following changes in Co concentrations. Large magneto-optical effects were induced at the band edge due to sp-d exchange interactions. However, magneto-optical activity was reduced in heavily doping concentrations above x ¼ 0.16 because of antiferromagnetic coupling between nearest-neighbor Co ions. The high magnetic activity at x ¼ 0.10 is related to competition between the complex Co-related configurations, such as singles, pairs, open and closed triples, in Co-doped ZnO layers. Magnetic Zn 1Àx Co x O therefore has an effective layer composition for applications of quantum barriers. V C 2013 AIP Publishing LLC. [http://dx.