We consider the dynamics of two-phase fluids, in particular the moving contact line, on a solid substrate. The dynamics are governed by the sharp-interface model consisting of the incompressible Navier-Stokes/ Stokes equations with the classical interface conditions, the Navier boundary condition for the slip velocity along the wall and a contact line condition which relates the dynamic contact angle of the interface to the contact line velocity. We propose an efficient numerical method for the model. The method combines a finite element method for the Navier-Stokes/Stokes equations on a moving mesh with a parametric finite element method for the dynamics of the fluid interface. The contact line condition is formulated as a timedependent Robin-type of boundary condition for the interface so it is naturally imposed in the weak form of the contact line model. For the Navier-Stokes equations, the numerical scheme obeys a similar energy law as in the continuum model but up to an error due to the interpolation of numerical solutions on the moving mesh. In contrast, for Stokes flows, the interpolation is not needed so we can prove the global unconditional stability of the numerical method in terms of the energy. Numerical examples are presented to demonstrate the convergence and accuracy of the numerical methods.