An object’s perceived readiness-for-action (e.g., its size, the degree of rotation from its canonical position, or the user’s viewpoint) can influence semantic knowledge retrieval. Yet, the organization of object knowledge may also be affected by body-specific sensorimotor experiences. Here, we investigated whether people’s history of performing motor actions with their hands influences the knowledge they store and retrieve about graspable objects. We compared object representations between healthy right- and left-handers (Experiment 1), and between unilateral stroke patients, whose motor experience was changed by impairment of either their right or left hand (Experiment 2). Participants saw pictures of graspable everyday items with the handles oriented toward either the left or right hand, and they generated the type of grasp they would employ (i.e., clench or pinch) when using each object, responding orally. In both experiments, hand dominance and object orientation interacted to predict response times. In Experiment 1, judgments were fastest when objects were oriented toward the right hand in right-handers, but not in left-handers. In Experiment 2, judgments were fastest when objects were oriented toward the left hand in patients who had lost the use of their right hand, even though these patients were right-handed prior to brain injury. Results suggest that at least some aspects of object knowledge are determined by motor experience, and can be changed by new patterns of motor experience. People with different bodily characteristics, who interact with objects in systematically different ways, form correspondingly different neurocognitive representations of the same common objects.