Assessment factors (AFs) are essential in the derivation of occupational exposure limits (OELs) and indoor air quality guidelines. The factors shall accommodate differences in sensitivity between subgroups, i.e., workers, healthy and sick people, and occupational exposure versus life-long exposure for the general population. Derivation of AFs itself is based on empirical knowledge from human and animal exposure studies with immanent uncertainty in the empirical evidence due to knowledge gaps and experimental reliability. Sensory irritation in the eyes and airways constitute about 30–40% of OELs and is an abundant symptom in non-industrial buildings characterizing the indoor air quality and general health. Intraspecies differences between subgroups of the general population should be quantified for the proposal of more ‘empirical’ based AFs. In this review, we focus on sensitivity differences in sensory irritation about gender, age, health status, and vulnerability in people, based solely on human exposure studies. Females are more sensitive to sensory irritation than males for few volatile substances. Older people appear less sensitive than younger ones. However, impaired defense mechanisms may increase vulnerability in the long term. Empirical evidence of sensory irritation in children is rare and limited to children down to the age of six years. Studies of the nervous system in children compared to adults suggest a higher sensitivity in children; however, some defense mechanisms are more efficient in children than in adults. Usually, exposure studies are performed with healthy subjects. Exposure studies with sick people are not representative due to the deselection of subjects with moderate or severe eye or airway diseases, which likely underestimates the sensitivity of the group of people with diseases. Psychological characterization like personality factors shows that concentrations of volatile substances far below their sensory irritation thresholds may influence the sensitivity, in part biased by odor perception. Thus, the protection of people with extreme personality traits is not feasible by an AF and other mitigation strategies are required. The available empirical evidence comprising age, lifestyle, and health supports an AF of not greater than up to 2 for sensory irritation. Further, general AFs are discouraged for derivation, rather substance-specific derivation of AFs is recommended based on the risk assessment of empirical data, deposition in the airways depending on the substance’s water solubility and compensating for knowledge and experimental gaps. Modeling of sensory irritation would be a better ‘empirical’ starting point for derivation of AFs for children, older, and sick people, as human exposure studies are not possible (due to ethical reasons) or not generalizable (due to self-selection). Dedicated AFs may be derived for environments where dry air, high room temperature, and visually demanding tasks aggravate the eyes or airways than for places in which the workload is balanced, while indoor playgrounds might need other AFs due to physical workload and affected groups of the general population.