Polarization Sensitive Optical Coherence Tomography (PSOCT) is a functional extension of Optical Coherence Tomography (OCT) that is sensitive to well-structured, birefringent tissue such as scars, smooth muscle and cartilage. In this work, we present a novel completely fiber based swept source PSOCT system using a fiber-optic rotary pullback catheter. This PSOCT implementation uses only passive optical components and requires no calibration while adding minimal additional cost to a standard structural OCT imaging system. Due to its complete fiber construction, the system can be made compact and robust, while the fiber-optic catheter allows access to most endoscopic imaging sites. The 1.5mm diameter endoscopic probe can capture 100 frames per second at pullback speeds up to 15 mm/s allowing rapid traversal of large imaging fields. We validate the PSOCT system with known birefringent tissues and demonstrate in vivo PSOCT imaging of human oral scar tissue.