A novel metamaterial cavity for the isolation enhancement of closely positioned dual‐polarized relay antenna arrays is proposed in this paper. The metamaterial cavity is composed of four sets of double‐layer complementary meta‐surface and metal reflector. Numerical and experimental results show that the isolation level of the same polarization ports for the proposed metamaterial cavity‐backed antennas can be improved by at most 10dB over the frequency band of 2.5–2.7 GHz, as well as the isolation of the cross polarization ports. More importantly, the metamaterial cavity‐backed antennas keep other good performance such as VSWR < 1.5 and a stable radiation pattern where the metamaterial cavity is loaded. The metamaterial cavity is suitable not only for the common antenna arrays, but also for the separate antenna arrays. The proposed design is easy to integrate with pico‐base station antennas, relay antennas and repeater antennas. © 2017 Wiley Periodicals, Inc. Microwave Opt Technol Lett 59:857–862, 2017