Residential interior design for people with special needs necessitates more specific research, especially with respect to physical abilities. Previous studies, as opposed to general people, have overlooked the nuanced requirements generated by age, gender, and living environment differences. This study aims to examine physical abilities across different ages, genders, and residencies and identify relationships between in-home mobility and physical abilities among people with special needs. A total of 384 participants (eight different types of disabilities) completed electronic questionnaires. Among young, adult, and older groups, physical abilities and self-support showed statistical differences. Males demonstrated higher physical ability levels but decreased hand–arm movements than females. Greater physical abilities and subjective health were found in urban residents compared to non-urban residents. Additionally, in-home mobility was positively associated with out-of-home mobility, physical abilities, independence, and subjective health (all p values < 0.05). Thus, physical abilities revealed disparities across ages, genders, and resident settings. Also, the greater the mobility inside the house, the more possibilities there are to enhance physical abilities both inside and outside the house, independence, and subjective health. Design recommendations are provided regarding layout, furniture, flooring, and lighting. The consideration of these specific requirements in residential interior design is encouraged.