Modern video surveillance systems often consist of multiple cameras capturing continuous videos. However, existing systems do not fully take advantage of shared scene semantics across multiple video streams, and hence querying the captured videos to find an event or object of interest can be a time-consuming task. In this paper, we propose a compact representation of objects and their relationships in the scene under surveillance, over time, and across multiple cameras, using a combined global spatio-temporal scene graph. The same objects that appear in multiple cameras are stored only once, reducing the storage required and speeding up the query when compared to storing the scene graph from each camera individually. Our experiments show that in a 5-camera system our proposed representation can speed up querying time by a factor of 3.9 times.