This paper presents a structural optimisation method in three-dimensional acoustic-elastic coupled problems. The proposed optimisation method finds an optimal allocation of elastic materials which reduces the sound level on some fixed observation points. In the process of the optimisation, configuration of the elastic materials is expressed with a level set function, and the distribution of the level set function is iteratively updated with the help of the topological derivative. The topological derivative is associated with state and adjoint variables which are the solutions of the acoustic-elastic coupled problems. In this paper, the acoustic-elastic coupled problems are solved by a BEM-FEM coupled solver, in which the fast multipole method (FMM) and a multi-frontal solver for sparse matrices are efficiently combined. Along with the detailed formulations for the topological derivative and the BEM-FEM coupled solver, we present some numerical examples of optimal designs of elastic sound scatterer to manipulate sound waves, from which we confirm the effectiveness of the present method.