Trypophobia refers to the visual discomfort experienced by some people when viewing clustered patterns (e.g., clusters of holes). Trypophobic images deviate from the 1/f amplitude spectra typically characterizing natural images by containing excess energy at mid-range spatial frequencies. While recent work provides partial support for the idea of excess mid-range spatial frequency energy causing visual discomfort when viewing trypophobic images, a full factorial manipulation of image phase and amplitude spectra has yet to be conducted. Here, we independently manipulated the phase and amplitude spectra of 31 Trypophobic images using a standard Fast Fourier Transform (FFT). Participants rated the four different versions of each image for levels of visual comfort, and completed the Trypophobia Questionnaire (TQ). Images having the original phase spectra intact (with either original or 1/f amplitude) explained the most variance in comfort ratings and were rated lowest in comfort. However, images with the original amplitude spectra but scrambled phase spectra were rated higher in comfort, with a smaller amount of variance in comfort attributed to the amplitude spectrum. Participant TQ scores correlated with comfort ratings only for images having the original phase spectra intact. There was no correlation between TQ scores and comfort levels when participants viewed the original amplitude / phase-scrambled images. Taken together, the present findings show that the phase spectrum of trypophobic images, which determines the pattern of small clusters of objects, plays a much larger role than the amplitude spectrum in determining visual comfort.