Under investigation in this paper is the higher-order Broer-Kaup(HBK) system, which describes the bidirectional propagation of long waves in shallow water. Via the standard truncated Painlevé expansion method, the residual symmetry of this system is derived. By introducing an appropriate auxiliary-dependent variable, the residual symmetry is successfully localized to Lie point symmetries. Via solving the initial value problems, the finite symmetry transformations are presented. However, the solution which obtained from the residual symmetry is a special group invariant solutions. In order to find more general solution of HBK system, we further generalize the residual symmetry method to the consistent tanh expansion (CTE) method and prove that the HBK system is CTE solvable, then the resonant soliton solutions and interaction solutions among different nonlinear excitations are obtained by the CET method.