After viewing a picture of an environment, our memory of it typically extends beyond what was presented—a phenomenon called boundary extension. But sometimes, memory errors show the opposite pattern—boundary contraction—and the relationship between these phenomena is controversial. We constructed virtual 3D environments, and created a series of views at different distances, from object close-ups to wide-angle indoor views, and tested for memory errors along this object-to-scene continuum. Boundary extension was evident for close-scale views, and transitioned parametrically to boundary contraction for far-scale views. However, this transition point was not tied to a specific position in the environment; instead, it tracked with judgments of the best looking view. We propose that boundary extension and contraction are in fact integrated phenomena, and we offer an account where competition between object-based and scene-based affordances determine whether a view will extend or contract in memory.