To mitigate the dependence on fossil fuels and the associated global warming issues, numerous studies have focused on the development of eco-friendly energy conversion devices such as polymer electrolyte membrane fuel cells (PEMFCs) that directly convert chemical energy into electrical energy. As one of the key components in PEMFCs, polymer electrolyte membranes (PEMs) should have high proton conductivity and outstanding physicochemical stability during operation. Although the perfluorinated sulfonic acid (PFSA)-based PEMs and some of the hydrocarbon-based PEMs composed of rationally designed polymer structures are found to meet these criteria, there is an ongoing and pressing need to improve and fine-tune these further, to be useful in practical PEMFC operation. Incorporation of organic/inorganic fillers into the polymer matrix is one of the methods shown to be effective for controlling target PEM properties including thermal stability, mechanical properties, and physical stability, as well as proton conductivity. Functionalization of organic/inorganic fillers is critical to optimize the filler efficiency and dispersion, thus resulting in significant improvements to PEM properties. This review focused on the structural engineering of functionalized carbon and silica-based fillers and comparisons of the resulting PEM properties. Newly constructed composite membranes were compared to composite membrane containing non-functionalized fillers or pure polymer matrix membrane without fillers.