The objectives of this study were to investigate the out‐of‐plane behavior of unreinforced masonry (URM) infill walls and to identify the effectiveness of tie systems that were proposed to enhance the infill stability under both in‐ and out‐of‐plane effects. This study examined three types of tie systems: a full‐length tie system (prototype), a staggered tie system and a half‐length tie system. For this purpose, four 1/4‐scale, single‐story, one‐bay masonry infilled reinforced concrete (RC) frame specimens with and without the proposed tie systems were sequentially tested under in‐plane static cyclic loading and out‐of‐plane dynamic excitation. In the experimental study, two series of shaking table tests were conducted to investigate the out‐of‐plane behavior of the infill walls. The first series compared the fundamental out‐of‐plane performance of three proposed tie systems, and the second series verified the effectiveness of the full‐length tie system to prevent infill out‐of‐plane failure under an excitation simulating the severe floor response in medium‐rise buildings. The test results indicated the following: (1) the presence of the tie systems caused minor differences in the in‐plane capacity and ductility of each specimen, (2) the effectiveness of the full‐length and staggered tie systems was superior to that of the half‐length tie system, and (3) the full‐length tie system not only prevented the infill out‐of‐plane failure but also significantly reduced the infill damage under realistic excitation conditions. The above research outcomes provide a useful contribution of the infill to upgrade the seismic resistance of buildings.