We present a detailed study of the structural dynamics, energetic and dynamical stability and thermal transport of the bismuth chalcogenides Bi2S3, Bi2Se3 and Bi2Te3 and their alloys. The differing activities of the Bi cation lone pairs in Bi2S3, Bi2Se3 and Bi2Te3 lead to competition between orthorhombic Pnma and rhombohedral R3 ̅ m phases, with the latter favored by heavier chalcogen atoms, while the reported non-ambient phases of Bi2Se3 and Bi2Te3 show phonon instabilities under ambient conditions. The Pnma structure has weaker chemical bonding and stronger phonon anharmonicity than the R3 ̅ m phase, resulting in an intrinsically lower lattice thermal conductivity. A thermodynamic model of Bi2(Se1-xSx)3 indicates that the R3 ̅ m structure is energetically favored only for low S content, but the stability window can potentially be extended at lower formation temperatures. Bi2(Se1-xTex)3 in the R3 ̅ m phase shows substantial deviation from ideal solid-solution behavior, due to a strong energetic preference for the Se and Te atoms to occupy the interior and exterior sites, respectively, in the constituent quintuple layers. Strain-field fluctuations induced by inhomogeneities in the chemical bonding are shown to play a significant role in determining the heat transport in the alloy systems, and chalcogen disorder away from the preferred symmetric structure is found to be an important factor in the reduced thermal conductivity of Bi2SeTe2 compared to the Bi2Te3 endpoint. The microscopic insight from these modelling studies provides new insight into the bismuth chalcogenides and their alloys, which may inform ongoing research to optimize the thermoelectric performance of these and related materials.