In order to account explicitly for the existence of long-periodic layered structures and the strong structural relaxations in the most common binary and ternary alloys of the Bi-Sb-Te-Se system, we have developed a one-dimensional cluster expansion (CE) based on first-principles electronic structure calculations, which accounts for the Bi and Sb bilayer formation. Excellent interlayer distances are obtained with a van der Waals density functional. It is shown that a CE solely based on pair interactions is sufficient to provide an accurate description of the ground-state energies of Bi-Sb-Te-Se binary and ternary systems without making the data set of ab initio calculated structures unreasonably large. For the binary alloys A 1−x Q x (A = Sb, Bi; Q = Te, Se), a ternary CE yields an almost continuous series of (meta)stable structures consisting of consecutive A bilayers next to consecutive A 2 Q 3 for 0 < x < 0.6. For x > 0.6, the binary alloy segregates into pure Q and A 2 Q 3 . The Bi-Sb system is described by a quaternary CE and is found to be an ideal solid solution stabilized by entropic effects at T = 0 K but with an ordered structure of alternating Bi and Sb layers for x = 0.5 at T = 0 K. A quintuple CE is used for the ternary Bi-Sb-Te system, where stable ternary layered compounds with an arbitrary stacking of Sb 2 Te 3 , Bi 2 Te 3 , and Te-Bi-Te-Sb-Te quintuple units are found, optionally separated by mixed Bi/Sb bilayers. Electronic properties of the stable compounds were studied taking spin-orbit coupling into account.