This paper presents an overview of representative up-to-date research and the authors’ own experimental results from tests of wall elements and a horizontally loaded timber-framed modular building. The research has been conducted in connection with the development of timber-based structures in recent years. In the present research, wall elements and modules of timber-frame construction with life-size dimensions were used. So far, these types of structures have mainly been tested in laboratories—especially with regard to anchoring and cyclic loading. An experimental testing was carried out on a natural scale in two stages based on the standard procedure described in EN 594. In the first stage, wall panels were tested. In the second stage, tests were carried out on a complete four-storey building. Dowel fasteners were used to fix the sheathing to the load-bearing wall structures. Additionally, the sheathing was glued to the timber frame of the walls. The same type of wall element was used for the construction of the tested building. Horizontal loads were applied at the height of the top of the walls in both stages. The building loads were applied in a direction perpendicular to the longitudinal axis of the modules. Based on test data, the stiffnesses of the wall panels and the whole building were derived, as well as the type of interaction between the modules and the influence of the walls on the spatial work of the building. On the basis of the conducted studies, both the stiffness of the walls in different configurations and the stiffness of the complete building were determined, as well as the nature of the interaction of neighbouring modules and the influence of wall connections on the 3D working of the building. The results show that the stiffness of the building in the horizontal plane in the direction of the applied load is almost twice as high as the sum of the stiffnesses of the building walls in the same direction.