Differences in mental health (MH) of users of distinct psychoactive substances have been shown. Both substance use (SU) and MH in users are influenced by stressful life events. This study compared MH parameters in distinct groups of substance users and evaluated the impact of stress factors on these outcomes. Data stem from the longitudinal Swiss Cohort Study on Substance Use Risk Factors (C-SURF) involving 4,475 young adult men. Distinct groups were created for the past 12 months' use of psychedelics, MDMA, psychostimulants, and cannabis. MH measurements (depressive symptoms, overall MH, perceived stress, life satisfaction) were used as outcome variables, while indicators of past family functioning and stressful life events served as covariates. The MH of psychedelics users was not significantly different from the no-drug-use group, whereas poorer MH was found in the other SU groups. Observed effects were influenced by the tested stress factors. The absence of association between use of psychedelics and worsening of MH deserves further investigation in male and female samples. Stressful life experiences must be considered when assessing the MH of users of illicit substances. These findings suggest that some men practice SU as self-medication to cope with life adversity.