The design of supported heterogeneous catalysts requires a detailed understanding of the structure and chemistry of the active surface. Although the chemical components of the active phase, support material, and process feed are typically considered to be the most important factors governing catalyst structure and performance, many common commercial supports contain trace impurities, which can have profound effects on catalyst properties. In this work, we study silica‐supported cobalt‐based catalysts, which are widely used in syngas conversion to value‐added products. Supported metallic Co is a commercial Fischer‐Tropsch catalyst, whereas Co2C has shown promise for the direct conversion of syngas to higher oxygenates. This study examines the effects of Na, a commonly detected support impurity and a frequently used promoter, on the structure and reactivity of Co and Co2C. We show that trace Na impurities significantly decrease catalyst activity of supported metallic Co, and that high Na concentrations result in Co2C formation and a loss in Fischer‐Tropsch activity. However, in Co2C catalysts, Na plays an important role in stabilizing the Co2C phase, but excess Na decreases catalyst activity. We use in situ X‐ray absorption spectroscopy to study Co2C formation and decomposition in the Na‐free catalyst under carburization and reaction conditions. The work reveals the importance of carefully controlling alkali metal content, particularly at trace levels, in catalyst design.