Among the various types of polymer electrolytes, gel polymer electrolytes have been considered as promising electrolytes for high-performance lithium and non-lithium batteries. The introduction of inorganic fillers into the polymer-salt system of gel polymer electrolytes has emerged as an effective strategy to achieve high ionic conductivity and excellent interfacial contact with the electrode. In this review, the detailed roles of inorganic fillers in composite gel polymer electrolytes are presented based on their physical and electrochemical properties in lithium and non-lithium polymer batteries. First, we summarize the historical developments of gel polymer electrolytes. Then, a list of detailed fillers applied in gel polymer electrolytes is presented. Possible mechanisms of conductivity enhancement by the addition of inorganic fillers are discussed for each inorganic filler. Subsequently, inorganic filler/polymer composite electrolytes studied for use in various battery systems, including Li-, Na-, Mg-, and Zn-ion batteries, are discussed. Finally, the future perspectives and requirements of the current composite gel polymer electrolyte technologies are highlighted.