Illicit cattle ranching and coca farming have serious negative consequences on the Colombian Amazon’s land systems. The underlying causes of these land activities include historical processes of colonization, armed conflict, and narco-trafficking. We aim to examine how illicit cattle ranching and coca farming are driving forest cover change over the last 34 years (1985–2019). To achieve this aim, we combine two pixel-based approaches to differentiate between coca farming and cattle ranching using hypothetical observed patterns of illicit activities and a deep learning algorithm. We found evidence that cattle ranching, not coca, is the main driver of forest loss outside the legal agricultural frontier. There is evidence of a recent, explosive conversion of forests to cattle ranching outside the agricultural frontier and within protected areas since the negotiation phase of the peace agreement. In contrast, coca is remarkably persistent, suggesting that crop substitution programs have been ineffective at stopping the expansion of coca farming deeper into protected areas. Countering common narratives, we found very little evidence that coca farming precedes cattle ranching. The spatiotemporal dynamics of the expansion of illicit land uses reflect the cumulative outcome of agrarian policies, Colombia’s War on Drugs, and the 2016 peace accord. Our study enables the differentiation of illicit land activities, which can be transferred to other regions where these activities have been documented but poorly distinguished spatiotemporally. We provide an applied framework that could be used elsewhere to disentangle other illicit land uses, track their causes, and develop management options for forested land systems and people who depend on them.