Background: The COVID-19 pandemic and its countermeasures may have had a significant impact on the psychological well-being of specific population subgroups. The present study investigated whether sexual minority men (defined here as attracted partly or exclusively to men) from an ongoing cohort study of young Swiss men experienced different psychological impacts, levels of substance use and addictive behaviors, and to which degree pre-existing vulnerabilities and participants experiences during the crisis might explain these differences. Methods: An ongoing cohort sample based on the general population of young Swiss men (mean age = 29.07 years; SD = 1.27) was assessed before and during the COVID-19 crisis for depression, stress, sleep quality, substance use and addictive behaviors. Additionally, during the crisis, we assessed its impact in form of fear, isolation and traumatic experiences. Potential associations between these outcomes and sexual orientation (sexual minority vs. heterosexual) were tested using linear regression models. It was additionally estimated to which degree these associations were attenuated if adjusted for differences in mental health, personality and socioeconomic status before the crisis, as well as the experience of the COVID-19 crisis (infection with the virus and changes to work situation). Results: Compared to heterosexual men, sexual minority men showed higher levels of psychological trauma (b = 0.37 [0.25, 0.49]), fear (b = 0.18 [0.06, 0.30]) and isolation (b = 0.32 [0.20, 0.44]) due to the COVID-19 pandemic as well as higher levels of depression (b = 0.31 [0.20, 0.41]) and lower sleep quality (b = -0.13 [-0.24, -0.02]) during the crisis. These differences were to a large degree explained by higher pre-crisis levels of mental health problems and the personality dimension of neuroticism-anxiety. Sexual minority men showed higher overall levels of substance use and addictive behaviors, but these differences were already present before the crisis. Conclusion: The COVID-19 crisis may have worsened pre-existing vulnerabilities in sexual minority men, leading to its greater psychological impact on them than on heterosexual men. Reducing minority stress due to sexual orientation may help not only to improve mental health among important proportions of the population but also to reduce their vulnerability to crises. Services offering psychological support to sexual minorities may need to be reinforced during crises.