Abstract Nonlinear responses to changing external pressures are increasingly studied in real-world ecosystems. However, as many of the changes observed by ecologists extend beyond the monitoring record, the occurrence of critical transitions, where the system is pushed from one equilibrium state to another, remains difficult to detect. Paleo-ecological records thus represent a unique opportunity to expand our temporal perspective to consider regime shifts and critical transitions, and whether such events are the exception rather than the rule. Yet, sediment core records can be affected by their own biases, such as sediment mixing or compression, with unknown consequences for the statistics commonly used to assess regime shifts, resilience, or critical transitions. To address this shortcoming, we developed a protocol to simulate paleolimnological records undergoing regime shifts or critical transitions to alternate states and tested, using both simulated and real core records, how mixing and compression affected our ability to detect past abrupt shifts. The smoothing that is built into paleolimnological data sets apparently interfered with the signal of rolling window indicators, especially autocorrelation. We thus turned to time-varying autoregressions (online dynamic linear models, DLMs; and time-varying autoregressive state-space models, TVARSS) to evaluate the possibility of detecting regime shifts and critical transitions in simulated and real core records. For the real cores, we examined both varved (annually laminated sediments) and non-varved cores, as the former have limited mixing issues. Our results show that state-space models can be used to detect regime shifts and critical transitions in some paleolimnological data, especially when the signal-to-noise ratio is strong. However, if the records are noisy, the online DLM and TVARSS have limitations for detecting critical transitions in sediment records.