Overlay is a notable user interface feature in the Android system, which allows an app to draw over other apps' windows. While overlay enhances user experience and allows concurrent app interaction, it has been extensively abused for malicious purposes, such as "tapjacking", leading to so-called overlay attacks. In order to combat this threat, Google introduced a dedicated window flag SYSTEM_FLAG_HIDE_NON_SYSTEM_OVERLAY_WINDOWS to protect critical system apps' windows against overlay attacks. Unfortunately, the adequacy of such protection in the Android system remains unstudied, with a noticeable absence of clear usage guidelines.To bridge the gap, in this paper, we conduct the first systematic study on the unprotected windows of system apps against overlay attacks. We propose a comprehensive guideline and then design and develop a new tool named OverlayChecker to identify the missing protections in Android system apps. To verify the uncovered issues, we also design and create Proof-of-Concept apps. After applying OverlayChecker to 8 commercial Android systems on 4 recently released Android versions, we totally discovered 49 vulnerable system apps' windows. We reported our findings to the mobile vendors, including Google, Samsung, Vivo, Xiaomi, and Honor. At the time of writing, 15 of them have been confirmed. 5 CVEs have been assigned, and 3 of them are rated high severity. We also received bug bounty rewards from these mobile vendors.