The Generalized Method of Moments is a technique to discretize integral equations that permits integration of different types of basis functions as well as different geometric descriptions using a partition of unity framework. While accuracy and efficacy of the method have been demonstrated, the integration quadratures required to compute the inner-products are often high as they have to respect spatial variation of the integrand. To overcome this problem, we introduce an interior penalty function method within the GMM framework. The penalty formulation yields solutions that are smooth and accurate both in surface currents and fields with significantly lower numerical quadrature orders than would be required for the uncompensated operator. To demonstrate and analyze the method, we conduct an analytical and numerical investigation of the properties of the penalty method applied to the 2D TE z Electric Field Integral Equation operator.