Anion substitution with bismuth (Bi) in III-V semiconductors is an effective method for experimental engineering of the band gap Eg at low Bi concentrations (≤ 2%), in particular in gallium arsenide (GaAs). The inverse Bi-concentration dependence of Eg has been found to be linear at low concentrations x and dominated by a valence band-defect level anticrossing between As and Bi occupied p levels. Predictive models for the valence band hybridization require a first-principle understanding which can be obtained by density functional theory with the main challenges being the proper description of Eg and the spin-orbit coupling. By using an efficient method to include these effects, it is shown here that at high concentrations Eg is modified mainly by a Bi-Bi p orbital interaction and by the large Bi atom-induced strain. In particular, we find that at high concentrations the Bi-Bi interactions depend strongly on model periodic cluster configurations, which is not captured by tight-binding models. Averaging over various configurations supports the defect level broadening picture. This points to the role of different atomic configurations obtained by varying the experimental growth conditions in engineering arsenide band gaps, in particular for telecommunication laser technology.