In the last decade, the behavioral sciences have described the phenomenon of trypophobia, which is the discomfort felt by some individuals when viewing images containing clusters of bumps or holes. One evolutionary hypothesis for this phenomenon is that this visual discomfort represents an adaptation which helps organisms avoid skin disease and/or ectoparasites. Even though trypophobic imagery and disease imagery are both examples of visual textures, to date there has been no comparison of the visual discomfort elicited by these two kinds of textures within the larger context of the visual comfort elicited by natural texture imagery. In the present study, we administered the Trypophobia Questionnaire (TQ) and recorded the visual comfort ratings elicited by a large set of standard natural texture images, including several trypophobic and skin disease images. We found that while all observers find skin diseases uncomfortable to view, only those scoring high on the TQ rated trypophobic imagery equally uncomfortable. Comparing the ratings of both high-TQ and low-TQ populations to the standard textures, we find remarkably consistent rank-order preferences, with both populations preferring textures which are darker, less dense, and more regular, and disliking textures which are brighter, more complex, and more irregular, with the most unpleasant textures (as rated by both groups) exhibiting qualitative similarities to trypophobic imagery. However, we find that low-level image statistics which have been previously shown to affect visual comfort are poor predictors of the visual comfort elicited by natural textures, and greatly under-predict the visual discomfort elicited by either trypophobic or disease imagery. Our results suggest that a full understanding of the visual comfort elicited by natural textures, including those arising from skin disease, will ultimately depend upon a better understanding of cortical areas specialized for the perception of surface and material properties, and how these visual regions interact with emotional brain areas to evoke appropriate behavioral responses, like disgust.