To improve the shear performance of assembly connections in precast RC shear wall structures, a novel polygon‐key assembly connection (PKA connection) was proposed. Shear tests were carried out to study the shear performance of the PKA connections, specimens' parameters considered in the tests consist of polygon‐key inclination angles, polygon‐key heights, and axial compression ratios. Shear capacity, stiffness, and failure modes of the PKA connections were investigated. Furthermore, theoretical analysis of the PKA connection was performed, equations of PKA connection's shear capacity were deduced, and the theoretical analysis results were verified by the test results. The results show that the polygon‐keys can effectively improve the shear capacity of assembly connection of precast RC shear wall structure. Compared with the parallel assembly connection, peak shear capacity of the PKA connection can be enhanced by more than 15%. When polygon‐key's inclination angle is within the range of 45–90°, the peak loads of PKA connections change slightly. When polygon‐key's height is within the range of 5–7 cm, the cracking load and peak load of the PKA connection change slightly. With an increase in axial compression ratio, the shear capacity of PKA connection gradually increases, and the peak displacement changes slightly. Based on the shear‐friction theory, the calculated PKA connection's peak loads are in good agreement with the test results.