In this paper, a kind of node_face frictional contact FM-BEM penalty function method is presented for 3D elastic frictional contact nonlinear problems. According to the principle of minimum potential energy, nonpenetrating constraints are introduced into the elastic frictional contact system as a penalty term. By using the least square method and penalty function method, an optimization mathematical model and a mathematical programming model with a penalty factor are established for the node_face frictional contact nonlinear system. For the two models, a penalty optimization IGMRES (m) algorithm is proposed, and the influences of different penalty factors on the solution of the whole system are analyzed. Finally, a numerical simulation is carried out for two elastic frictional contact objects, and some important results including displacements, pressures, friction forces, and friction slips in the contact area are presented. Theoretical analysis and numerical experiment show that the newly presented FM-BEM penalty function method not only is efficient and practical but also has much superiority. It is easy to implement, and it is fast convergent with good stability.