In physics, experiments ultimately inform us about what constitutes a good theoretical model of any physical concept: physical space should be no exception. The best picture of physical space in Newtonian physics is given by the configuration space of a free particle (or the center of mass of a closed system of particles). This configuration space (as well as phase space) can be constructed as a representation space for the relativity symmetry. From the corresponding quantum symmetry, we illustrate the construction of a quantum configuration space, similar to that of quantum phase space, and recover the classical picture as an approximation through a contraction of the (relativity) symmetry and its representations. The quantum Hilbert space reduces into a sum of one-dimensional representations for the observable algebra, with the only admissible states given by coherent states and position eigenstates for the phase and configuration space pictures, respectively. This analysis, founded firmly on known physics, provides a quantum picture of physical space beyond that of a finite-dimensional manifold and provides a crucial first link for any theoretical model of quantum space-time at levels beyond simple quantum mechanics. It also suggests looking at quantum physics from a different perspective.