Elastomers are made of chain-like molecules to form networks that can sustain large deformation. Rubbers are thermosetting elastomers that are obtained from irreversible curing reactions. Curing reactions create permanent bonds between the molecular chains. On the other hand, thermoplastic elastomers do not need curing reactions. Incorporation of appropriated filler particles, as has been practiced for decades, can significantly enhance mechanical properties of elastomers. However, there are fundamental questions about polymer matrix composites (PMCs) that still elude complete understanding. This is because the macroscopic properties of PMCs depend not only on the overall volume fraction (ϕ) of the filler particles, but also on their spatial distribution (i.e., primary, secondary, and tertiary structure). This work aims at reviewing how the mechanical properties of PMCs are related to the microstructure of filler particles and to the interaction between filler particles and polymer matrices. Overall, soft rubbery matrices dictate the elasticity/hyperelasticity of the PMCs while the reinforcement involves polymer–particle interactions that can significantly influence the mechanical properties of the polymer matrix interface. For ϕ values higher than a threshold, percolation of the filler particles can lead to significant reinforcement. While viscoelastic behavior may be attributed to the soft rubbery component, inelastic behaviors like the Mullins and Payne effects are highly correlated to the microstructures of the polymer matrix and the filler particles, as well as that of the polymer–particle interface. Additionally, the incorporation of specific filler particles within intelligently designed polymer systems has been shown to yield a variety of functional and responsive materials, commonly termed smart materials. We review three types of smart PMCs, i.e., magnetoelastic (M-), shape-memory (SM-), and self-healing (SH-) PMCs, and discuss the constitutive models for these smart materials.