In this paper we present a Stackelberg differential game to study the dynamic interaction between a polluting firm and a regulator who sets pollution limits overtime. At each time, the firm settles emissions taking into account the fine for non-compliance with the pollution limit, and balances current costs of investments in a capital stock which allows for future emission reductions. We derive two main results. First, we show that the optimal pollution limit decreases as the capital stock increases, while both emissions and the level of noncompliance decrease. Second, we find that offering fine discounts in exchange for firm's capital investment is socially desirable. We numerically obtain the optimal value of such discount, which crucially depends on the severity of the fine. In the limiting scenario with a very large severity of the fine, the optimal discount implies that no penalties are levied, since the firm shows adequate adaptation progress through capital investment.