Environmental noise from transportation or industrial infrastructure typically has a broad frequency range. Different sources may have disparate acoustical characteristics, which may in turn affect noise annoyance. However, knowledge of the relative contribution of the different acoustical characteristics of broadband noise to annoyance is still scarce. In this study, the subjectively perceived short-term (acute) annoyance reactions to different broadband sounds (namely, realistic outdoor wind turbine and artificial, generic sounds) at 40 dBA were investigated in a controlled laboratory listening experiment. Combined with the factorial design of the experiment, the sounds allowed for separation of the effects of three acoustical characteristics on annoyance, namely, spectral shape, depth of periodic amplitude modulation (AM), and occurrence (or absence) of random AM. Fifty-two participants rated their annoyance with the sounds. Annoyance increased with increasing energy content in the low-frequency range as well as with depth of periodic AM, and was higher in situations with random AM than without. Similar annoyance changes would be evoked by sound pressure level changes of up to 8 dB. The results suggest that besides standard sound pressure level metrics, other acoustical characteristics of (broadband) noise should also be considered in environmental impact assessments, e.g., in the context of wind turbine installations.