Rectangularity and perpendicularity of contours are important properties of 3D shape for thevisual system and the visual system can use them as a priori constraints for perceiving shape veridically.The present article provides a comprehensive review of prior studies of the perception of rectangularityand perpendicularity and it discusses their effects on 3D shape perception from both theoretical andempirical approaches. It has been shown that the visual system is biased to perceive a rectangular 3Dshape from a 2D image. We thought that this bias might be attributable to the likelihood of arectangular interpretation but this hypothesis is not supported by the results of our psychophysicalexperiment. Note that the perception of a rectangular shape cannot be explained solely on the basis ofgeometry. A rectangular shape is perceived from an image that is inconsistent with a rectangularinterpretation. To address this issue, we developed a computational model that can recover arectangular shape from an image of a parallelopiped. The model allows the recovered shape to beslightly inconsistent so that the recovered shape satisfies the a priori constraints of maximumcompactness and minimal surface area. This model captures some of the phenomena associatedwith the perception of the rectangular shape that were reported in prior studies. This finding suggeststhat rectangularity works for shape perception by incorporating it with some additional constraints.