Wearing face masks is considered as one of the infection prevention and control options for respiratory viruses (e.g., SARS-CoV-2) that acts by blocking virus-laden aerosols. It is generally thought that aerosol blockage occurs when air passes through the face mask fabric. We calculated air flows through face masks and through peripheral leakages, based on reported breathing resistance values of face masks (FFP/N95, surgical masks, and cloth masks) and found that most of the inhaled and exhaled air passes through these peripheral leakages. Nevertheless, face masks remain effective as an infection prevention option, because additional calculations showed that the majority of aerosol volume cannot follow the tortuous path of air around the face mask. The filtering efficiency through the peripheral leakages can be described as a function of breathing conditions, vocal activities, the leakage geometry and tortuous pathway, aerosol properties (diameter, composition) and ambient conditions (e.g., evaporation, ventilation). Inclusion of these parameters explains the asymmetric filtering behavior of face masks, i.e., the risk of infection from person A to person B does not necessarily equal the risk of infection from person B to person A. Our findings explain thus why masking an infectious person is more effective than masking an exposed person. Establishing that the tortuous pathway of air around the face mask is the sole contributor to face mask efficiency opens new opportunities for designing safer face masks.