A polarization sensitive endoscopic optical frequency domain imaging (PS-OFDI) system with a motorized distal scanning catheter is demonstrated. It employs a passive polarization delay unit to multiplex two orthogonal probing polarization states in depth, and a polarization diverse detection unit to detect interference signal in two orthogonal polarization channels. Per depth location four electro-magnetic field components are measured that can be represented in a complex 2x2 field matrix. A Jones matrix of the sample is derived and the sample birefringence is extracted by eigenvalue decomposition. The condition of balanced detection and the polarization mode dispersion are quantified. A complex field averaging method based on the alignment of randomly pointing field phasors is developed to reduce speckle noise. The variation of the polarization states incident on the tissue due to the circular scanning and catheter sheath birefringence is investigated. With this system we demonstrated imaging of ex vivo chicken muscle, in vivo pig lung and ex vivo human lung specimens.