An evaluation of the failures of coal under variable conditions in relation to the stability of coal pillars is an important issue for coal mining. This paper presents the results of an experimental investigation of the mechanical behavior of brittle coal under plane-strain biaxial stress state (PSBSS) compression. The synthetic specimens were meant to test the ductile properties. Modified surface instability detection apparatus was setup to carry out the tests. The results show that the stress-strain curves of coal and synthetic specimens under the PSBSS can be divided into four typical stages, namely original microcrack closure, elastic deformation, sudden stress drop, and residual behavior. The stress-strain curve of coal under PSBSS compression showed periodic stress drops during the post-peak phase, while the curve of the synthetic specimen presented a moderate decrease during the post-peak stage. The remarkable residual strengths of the coal and composites can be observed using these curves, which is vital to the stability of specimens after the failure strength is exceeded. Strain rates have a significant effect on the strength of coal samples under PSBSS compression. Specifically, a higher strain rate indicates a greater peak strength. The failure modes for coal and synthetic specimens are different. During PSBSS compressive tests, coal showed a split failure under different strain rates. For flexible composites, the failure pattern is conjugate shear failure. The study indicated that the Modified Lade and Modified Wiebols-Cook criteria are competent for estimating the defined strength coefficient, which is a ratio proposed to estimate the PSBSS peak strength.