As a major socio-historical event affecting different aspects of life, the COVID-19 pandemic presents a unique opportunity to study how different population groups adapt. We investigate the impact of this crisis on the evolution of perceived stress in the short and medium term in Switzerland, using data of the Swiss Household Panel from 2016 to early 2021, which include annual measures of perceived stress and a study between waves, conducted in May and June 2020 at the end of the first semi-lockdown. Using the longitudinal structure of the data with pre-crisis measurements, we estimate pooled OLS, fixed effects and first difference models, which include socio-demographic variables, life events, socio-economic status, work-related variables, stress-reducing resources and restrictions in place. Results for the overall population show a continuous increase in stress levels between 2016 and 2019 and a stress reduction right after the first semi-lockdown followed by a return to pre-pandemic levels. Privileged groups with higher levels of stress before the pandemic were most likely to reduce perceived stress. Characteristics related to more favourable trajectories include stable or improved financial situations and high levels of education (short-term effects), and high-pressure jobs and working hours (short- and medium-term effects). Our analyses reveal the importance of resources, such as social relations and work–life balance, to individuals’ management of the effects of the pandemic. Our results show that the effects of the pandemic on perceived stress are context-specific. They underline the importance of longitudinal analyses to understand the complexity of vulnerability and adaptation processes.