Second harmonic generation (SHG) microscopy has emerged as a powerful technique for visualizing collagen organization within tissues. Amongst the many advantages of SHG is its sensitivity to collagen nanoscale organization, and its presumed sensitivity to the relative out of plane polarity of fibrils. Recent results have shown that circular dichroism SHG (CD-SHG), a technique that has been commonly assumed to reveal the relative out of plane polarity of collagen fibrils, is actually insensitive to changes in fibril polarity. However, results from another research group seem to contradict this conclusion. Both previous results have been based on SHG imaging of collagen fibrils within tissues, therefore, to gain a definitive understanding of the sensitivity of SHG to relative out of plane polarity, the results from individual fibrils are desirable. Here we present polarization resolved SHG microscopy (PSHG) data from individual collagen fibrils oriented out of the image plane by buckling on an elastic substrate. We show through correlation with atomic force microscopy measurements that SHG intensity can be used to estimate the out of plane angle of individual fibrils. We then compare the sensitivity of two PSHG techniques, CD-SHG and polarization-in, polarization-out SHG (PIPO-SHG), to the relative out of plane polarity of individual fibrils. We find that for single fibrils CD-SHG is insensitive to relative out of polarity and we also demonstrate the first direct experimental confirmation that PIPO-SHG reveals the relative out of plane polarity of individual collagen fibrils.