Solid-state polymer electrolytes (SPEs) have great processing flexibility and electrode–electrolyte contact and have been employed as the promising electrolytes for lithium metal batteries. Among them, poly(ethylene oxide) (PEO)-based SPEs have attracted widespread attention because of easy synthesis, low mass density, good mechanical stability, low binding energy with lithium salts, and excellent mobility of charge carriers. In order to overcome the low room-temperature ionic conductivity and the poor thermodynamic stability in high-voltage devices (>4.2 V) of the PEO materials, composition modulations by incorporating PEO with inorganic and/or organic components have been designed, which could effectively enable the applications of PEO-based SPEs with widened electro-stable voltage ranges. In this mini review, we describe recent progresses of several kinds of PEO composite structures for SPEs, and we compare the synthesis strategies and properties of these SPEs in lithium batteries. Further developments and improvements of the PEO-based materials for building better rechargeable batteries are also discussed.