In acoustics, the boundary element method (BEM) is much more common compared to elasticity. This is driven by the applications, which are in acoustics very often radiation problems causing trouble in finite element or finite volume methods. Nevertheless, for a time domain calculation, still an efficient BE formulations is lacking. We consider the time domain BEM for the homogeneous wave equation with vanishing initial conditions and given Dirichlet boundary conditions. The generalized convolution quadrature method (gCQ) developed by Lopez‐Fernandez and Sauter is used for the temporal discretisation. The spatial discretisation is done classically. Essentially, the gCQ requires to establish boundary element matrices of the corresponding elliptic problem in Laplace domain at several complex frequencies. Consequently, an array of system matrices is obtained. This array of system matrices can be interpreted as a three‐dimensional array of data, which should be approximated by a data‐sparse representation. The adaptive cross approximation (ACA) can be generalized to handle these three‐dimensional data arrays. Adaptively, the rank of the three‐dimensional data array is increased until a prescribed accuracy is obtained. On a pure algebraic level, it is decided whether a low‐rank approximation of the three‐dimensional data array is close enough to the original matrix. Hierarchical matrices are used in the two spatial dimensions and the third dimension are the complex frequencies. Hence, the algorithm makes not only a data sparse approximation in the two spatial dimensions but detects adaptively how much frequencies are necessary for which matrix block.