The primary goal of this paper is to study a nonlinear complementarity system (NCS, for short) with a nonlinear and nonhomogeneous partial differential operator and mixed boundary conditions, and a simultaneous distributed-boundary optimal control problem governed by (NCS), respectively. First, we formulate the weak formulation of (NCS) to a mixed variational inequality with double obstacle constraints (MVI, for short), and prove the existence and uniqueness of solution to (MVI). Then, a power penalty method is applied to (NCS) for introducing an approximating mixed variational inequality without constraints (AMVI, for short). After that, a convergence result that the unique solution of (MVI) can be approached by the unique solution of (AMVI) when a penalty parameter tends to infinity, is established. Moreover, we explore the solvability of the simultaneous distributed-boundary optimal control problem described by (MVI), and consider a family of approximating optimal control problems driven by (AMVI). Finally, we provide a result on asymptotic behavior of optimal controls, system states and minimal values to approximating optimal control problems.