Prior knowledge can facilitate or distort new episodic memories, depending on their 2 alignment. However, it remains unclear how the structure of semantic memory explains 3 systematic variation in how new memories are formed and retrieved. We aimed to 4 quantify distortions in memory by examining how category membership and typicality 5 bias new memories. Across four experiments, participants encoded and retrieved 6 image-location associations. Most members of a category (e.g. birds) were located near 7 each other, such that participants could learn locations of categories as they encoded 8 specific image locations. Critically, some typical and atypical category members were in 9 random locations. We decomposed location memory into two measures: error, a measure of episodic specificity, and bias towards other category members, a measure of the influence of prior knowledge. First, we found that location memory was more accurate for images that were spatially consistent with their category membership.Second, when images were spatially inconsistent, retrieval of typical category members was more biased towards other category members relative to atypical ones. These effects replicated across three experiments, disappeared when images were not arranged by category, and were stronger than effects observed with images arranged by visual similarity rather than category membership. Our observations provide compelling evidence that memory is a reconstruction of multiple sources of information, integrating memory for specific events with relevant semantic knowledge. Furthermore, systematic differences in the magnitude of this integration suggest that the organization of semantic memory can govern the extent of distortion in new episodic memories.