Building-level characteristics are structural factors largely beyond the control of those who live in them. We explored whether building-level characteristics and indoor allergens in the household are related. We examined the relationship between building-level characteristics and indoor allergens: dust mite, cat, cockroach, and mouse. Building-level characteristics measured were presence of pests (seeing cockroaches and rodents), building type (public housing, buildings zoned commercially and residentially, and building size), and building condition (building age and violations). Allergen cutpoints were used for categorical analyses and defined as follows: dust mite: 90.25 μg/g; cat: 91 μg/g; cockroach: 91 U/g; mouse: 91.6 μg/g. In fully adjusted linear analyses, neither dust mite nor cat allergen were statistically significantly associated with any building-level characteristics. Cockroach allergen was associated with the presence of cockroaches (2.07; 95% CI, 1.23, 3.49) and living in public housing (2.14; 95% CI, 1.07, 4.31). Mouse allergen was associated with the presence of rodents (1.70; 95% CI, 1.29, 2.23), and building size: living in a low-rise (G8 floors; 0.60; 95% CI, 0.42, 0.87) or high-rise (8+floors; 0.50; 95% CI, 0.29, 0.88; compared with house/duplex). In fully adjusted logistic analyses, cat allergen was statistically significantly associated with living in a high-rise (6.29; 95% CI, 1.51, 26.21; compared with a house/duplex). Mouse allergen was associated with living in public housing (6.20; 95% CI, 1.01, 37.95) and building size: living in a low-rise (0.16; 95% CI, 0.05, 0.52) or high-rise (0.06; 95% CI, 0.01, 0.50; compared with a house/duplex). Issues concerning building size and public housing may be particularly critical factors in reducing asthma morbidity. We suggest that future research explore the possible improvement of these factors through changes to building code and violations adherence, design standards, and incentives for landlords.