Variation in political preferences is increasingly understood to stem from both environmental influences and genetics. Research has argued that a possible pathway for genetic effects on ideology is via cognitive performance, showing a genetic overlap between the traits. Yet, an unambiguous link between cognitive performance and economic policy preferences remains elusive, with results ranging from negative to positive effects on economic conservatism. In this study, I argue that this heterogeneity reflects an underlying gene-by-environment interaction. I depart from the assumption that cognitive performance, rather than being directly connected to a set of policy preferences, influences the capacity to correctly optimize those preferences. Combined with insights from standard models in political economics, this leads to the proposal that genetics associated with cognitive performance should cause more left-wing economic preferences if you grow up in relative poverty, but more right-wing preferences if you grow up in relative affluence. Utilizing variation in a polygenic index of cognitive performance within dizygotic twin pairs, coupled with unique register data on economic conditions for both the twins, their parents, and their childhood neighborhood, I show that the effect of the PGI on a fine-grained measure of economic conservatism is zero on average, but indeed sizable and sign-discordant by class background. To my knowledge, this represents the first sign-discordant GxE finding for a socially relevant outcome, which has implications for future social science research using PGIs.