Open quantum systems (OQSs) cannot always be described with the Markov approximation, which requires a large separation of system and environment time scales. Here, we give an overview of some of the most important techniques available to tackle the dynamics of an OQS beyond the Markov approximation. Some of these techniques, such as master equations, Heisenberg equations and stochastic methods, are based on solving the reduced OQS dynamics, while others, such as path integral Monte Carlo or chain mapping approaches, are based on solving the dynamics of the full system. We emphasize the physical interpretation and derivation of the various approaches, explore how they are connected and examine how different methods may be suitable for solving different problems.