Semantic memory is described as the storage of knowledge, concepts, and information that is common and relatively consistent across individuals (e.g., memory of what is a cup). These memories are stored in multiple sensorimotor modalities and cognitive systems throughout the brain (e.g., how a cup is held and manipulated, the texture of a cup's surface, its shape, its function, that is related to beverages such as coffee, and so on). Our ability to engage in purposeful interactions with our environment is dependent on the ability to understand the meaning and significance of the objects and actions around us that are stored in semantic memory. Theories of the neural basis of the semantic memory of objects have produced sophisticated models that have incorporated to varying degrees the results of cognitive and neural investigations. The models are grouped into those that are (1) cognitive models, where the neural data are used to reveal dissociations in semantic memory after a brain lesion occurs; (2) models that incorporate both cognitive and neuroanatomical information; and (3) models that use cognitive, neuroanatomic, and neurophysiological data. This review highlights the advances and issues that have emerged from these models and points to future directions that provide opportunities to extend these models. The models of object memory generally describe how category and/or feature representations encode for object memory, and the semantic operations engaged in object processing. The incorporation of data derived from multiple modalities of investigation can lead to detailed neural specifications of semantic memory organization. The addition of neurophysiological data can potentially provide further elaboration of models to include semantic neural mechanisms. Future directions should incorporate available and newly developed techniques to better inform the neural underpinning of semantic memory models.