Penalty functions can be used to add constraints to systems of equations. In computational mechanics, stiffness-type penalties are the common choice. However, in dynamic applications stiffness penalties have the disadvantage that they tend to decrease the critical time step in conditionally stable time integration schemes, leading to increased CPU times for simulations. In contrast, inertia penalties increase the critical time step. In this paper, we suggest the simultaneous use of stiffness penalties and inertia penalties, which is denoted as the bipenalty method. We demonstrate that the accuracy of the bipenalty method is at least as good as (and usually better than) using either stiffness penalties or inertia penalties. Furthermore, for a number of finite elements (bar elements, beam elements and square plane stress/plane strain elements) we have derived ratios of the two penalty parameters such that their combined effect on the critical time step is neutral. The bipenalty method is thus superior to using stiffness penalties, because the decrease in critical time step can be avoided. The bipenalty method is also superior to using inertia penalties, since the constraints are realized with higher accuracy.