Outdoor recreation is increasing in intensity and space. Areas previously inaccessible are now being visited by ever-growing numbers of people, which increases human-wildlife encounters across habitats. This has raised concern among researchers and conservationists as, even in non-aggressive encounters, animals often perceive humans as predators and mount physiological and behavioural responses that can have negative consequences. However, despite all the research in recent decades, not many general patterns have emerged, especially at the level of populations, and many studies have yielded seemingly contradictory or inconclusive results. We argue that this is partly due to incomplete knowledge of the number and complexity of factors that may modulate the responses of animals. Thus, we aim to provide a conceptual approach intended to highlight the reasons that make it difficult to detect general patterns. We present a comprehensive compilation of factors modulating animal responses to humans at increasing levels (from sensory detection and immediate behavioural and physiological reactions, to changes in fitness and population trends), which may help understanding the uncertainty in the patterns. We observed that there are many modulating factors, which can be categorized as reflecting characteristics of the recreational activity itself (e.g. intensity of human presence), of the animals concerned (e.g. age or antipredatory strategy), and of the spatio-temporal context (e.g. habitat or timing of the encounter). Some factors appear to have non-linear and complex effects, which, if not considered, may lead to erroneous conclusions. Finally, we conclude that the difficulty in finding general patterns will be amplified at higher levels (i.e. at the level of populations), since as we proceed from one level to the next, the number of potential modulating factors accumulates, adding noise and obscuring direct associations between recreation and wildlife. More comprehensive knowledge about which (and how) factors affect animal responses across levels will certainly improve future research design and interpretation, and thus, our understanding of human recreational impacts on wildlife.