We investigated patterns and drivers of alcohol misuse and illicit drug use among young fisherfolk. We undertook this study in fishing communities on Koome Island, Lake Victoria, Uganda, from December 2017-July 2018. We conducted six group discussions with men (3) and women (3) and 33 in-depth interviews with: young people [users (n = 10); non-users (n = 2)], local leaders (n = 3), health workers (n = 2), parents (n = 5), alcohol/illicit drugs sellers/distributors (n = 5), law enforcement officers (n = 5). We sampled participants using purposive and snowball strategies. Interview themes included: knowledge, experiences and perceptions of alcohol use/illicit drug use, HIV risk behaviour and harm reduction. We mapped alcohol/illicit drug use outlets using a Geographic Information System to capture density, distribution and proximity to young people's homes. We coded and analysed qualitative data using thematic content analysis. Motivations for heavy drinking and illicit drug use were multifaceted and largely beyond individual control. Key contextual determinants included social norms around consumption (acceptability), price (affordability), and ease of purchase (availability). Prevention and harm reduction interventions to tackle alcohol misuse and illicit drug use should be aimed at the structural rather than individual level and must be conducted in tandem with strategies to control poverty and HIV.