Based on the two-dimensional (2D) elastic theory without enforcing any beam assumption, an efficient semi-analytical scaled boundary finite element method (SBFEM) is proposed to solve the bending and free vibration responses of composite laminated and sandwich beams under the mechanical load. The scaled center is placed at infinity, which produces the accurate result by discretizing only the longitudinal direction of the beam structure treated as a one-dimensional (1D) discretization problem. A new kind of 1D high-order spectral element shape functions with the advantages of high accuracy and superior convergence is introduced in SBFEM coordinate system to approximate the geometric model and corresponding variables. The principle of weighted residual in conjunction with the Green’s theorem are applied to obtain the SBFEM governing equation of each layer with respect to radial displacement fields. The solution of equation is indicated analytically by a matrix exponential function, which can be accurately solved by using the precise integration technique (PIT). Finally, an effective and simple stiffness matrix is obtained. By comparing two examples with the solutions based on the finite element method (FEM), the results show that the proposed method has good accuracy and rapid convergence with only a few meshes. The numerical examples are given to investigate the parametric effects of the stacking sequence, thickness ratio, boundary condition, and load form on the variation of the displacement, stress and natural frequency. The results validate that the present technique is also applicable to the complex beam structure with softcore layer inside.