Global change has increased inorganic nitrogen (N) and dissolved organic carbon (DOC; i.e., "browning") inputs to northern hemisphere boreal lakes. However, we do not know how phytoplankton in nutrient poor lake ecosystems of different DOC concentration respond to increased N availability. Here, we monitored changes in phytoplankton production, biomass and community composition in response to whole lake inorganic N fertilization in six boreal unproductive Swedish lakes divided into three lake pairs (control, N enriched) at three DOC levels (low, medium, high), with one reference year (2011) and 2 impact yr (2012, 2013). We found that phytoplankton biomass and production decreased with DOC concentration before N fertilization. Further, phytoplankton community composition also differed with respect to DOC, with a dominance of non-flagellated autotrophs at low DOC towards an increasing dominance of flagellated autotrophs with increased lake DOC concentration. The N fertilization increased phytoplankton biomass and production in all lakes, but did not affect phytoplankton community composition. However, the net response in biomass and production to N fertilization declined with increasing DOC, implying that the lake DOC concentration is critical in order to infer phytoplankton responses to N fertilization, and that the system switches from being primarily nutrient limited to becoming increasingly light limited with increased DOC concentration. In conclusion, our results show that browning will reduce phytoplankton production and biomass and influence phytoplankton community composition, whereas increased inorganic N loadings from deposition, forestry or other land use will primarily enhance phytoplankton biomass and production. Together, any change in the landscape that enhances inorganic N availability will increase phytoplankton production and biomass, but the positive effects of N will be much weaker or even neutralized in browner lakes as caused by light limitation.