Gillnets are the most commonly used fishing gear targeting black rockfish (Sebastes schlegeli) and fat greenling (Hexagrammos otakii) in the artificial reef area in coastal China. However, a minimum mesh size (MMS) has not yet been established for this fishery, which is unfavorable for sustainable development and effective management of fishery resources. We compared size selectivity and catch efficiency of gillnets of four commonly used mesh size (stretched mesh size; 40, 50, 60, and 70‐mm) in a typical artificial reef area in the Yellow Sea, China. Increasing the mesh size improved size selectivity of gillnets for black rockfish and fat greenling by reducing the proportion of undersized individuals. Further, changes in mesh size affected catch efficiency of target‐sized individuals. To protect juveniles, while maintaining catch efficiency for target‐sized fish, we recommend 60‐mm as the MMS for this fishery. Our study will contribute to the development of MMS regulations in reef fisheries management in China.