The GaAs1-xBix semiconductor alloy allows to achieve large bandgap reduction and enhanced spin-orbit splitting energy at dilute Bi quantities. The bismide is currently being developed for near-to midinfrared lasers, multi-junction solar cells, and photodetectors. In this structure-property relationship study of GaAsBi alloys, we report polarization dependent photoluminescence that reaches ratio up to 2.4 at room temperature. Polarization dependence is also presented using transmittance spectra, and also birefringence and linear dichroism. The optical anisotropy observations agree with predictions of point symmetry reduction in the CuPtB-type ordered GaAsBi phase. The structural ordering is investigated experimentally from the atomic-scale in molecular-beam epitaxy (MBE) grown samples on exact and miscut (001) GaAs substrates, as well as a sample composed of anti-phase domains in which the ordering axes are rotated by right angles. Since the conditions stabilizing the CuPtB ordered phase fall within the typical MBE growth regime of dilute bismides, the optical anisotropy in GaAsBi alloys is expected to be ubiquitous. These findings are important for the future development of GaAsBibased optoelectronics and also provide new means to analyze structurally complex bismide alloys.