Quantifying the potential for abrupt non-linear changes in ecological communities is a key managerial goal, leading to a significant body of research aimed at identifying indicators of approaching regime shifts. Most of this work has built on the theory of bifurcations, with the assumption that critical transitions are a common feature of complex ecological systems. This has led to the development of a suite of often inaccurate early warning signals (EWSs), with more recent techniques seeking to overcome their limitations by analysing multivariate time series or applying machine learning. However, it remains unclear whether regime shifts and/or critical transitions are common occurrences in natural systems, and - if they are present - whether classic and second-generation EWS methods predict rapid community change. Here, using multi-trophic data on nine lakes from around the world, we both identify the type of transition a lake is exhibiting, and the reliability of classic and second generation EWSs methods to predict whole ecosystem change. We find few instances of critical transitions in our lake dataset, with different trophic levels often expressing different forms of abrupt change. The ability to predict this change is highly technique dependant, with multivariate EWSs generally classifying correctly, classical rolling window univariate EWSs performing not better than chance, and recently developed machine learning techniques performing poorly. Our results suggest that predictive ecology should start to move away from the concept of critical transitions and develop methods suitable for predicting change in the absence of the strict bounds of bifurcation theory.