Two-dimensional materials can be combined by placing individual layers on top of each other, so that they are bound only by their van der Waals interaction. The sequence of layers can be chosen arbitrarily, enabling an essentially atomic-level control of the material and thereby a wide choice of properties along one dimension. However, simultaneous control over the structure in the in-plane directions is so far still rather limited. Here, we combine spatially controlled modifications of 2D materials, using focused electron irradiation or electron beam induced etching, with the layer-by-layer assembly of van der Waals heterostructures. The presented assembly process makes it possible to structure each layer with an arbitrary pattern prior to the assembly into the heterostructure. Moreover, it enables a stacking of the layers with accurate lateral alignment, with an accuracy of currently 10 nm, under observation in an electron microscope. Together, this enables the fabrication of almost arbitrary 3D structures with highest spatial resolution.