We show the potential of cross-sectional scanning tunneling microscopy to address structural properties of dilute III-V bismides by investigating Bi:InP. Bismuth atoms down to the second monolayer below the {110} InP surfaces, which give rise to three classes of distinct contrast, are identified with the help of density functional theory calculations. Based on this classification, the pair-correlation function is used to quantify the ordering of Bi atoms on the long range. In a complementary short-ranged study, we investigate the Bi ordering at the atomic level. An enhanced tendency for the formation of first-nearest-neighbor Bi pairs is found. In addition, the formation of small Bi clusters is observed whose geometries appear to be related to strong first-nearest-neighbor Bi pairing. We also identify growth related crystal defects, such as In vacancies, P antisites, and Bi antisites.