Intense stress increases substance use (SU). However, little is known about the extent to which distinctive forms of stress should be weighted with regard to their effects on SU. This study aimed to determine whether family-related stress factors (FSF) influenced SU in a different way than external stress factors (ESF). Data was drawn from a Swiss cohort study on SU risk factors (C-SURF), involving 5,308 young adult men. Twelve month use of alcohol and of illicit substances was assessed. FSF and ESF for the time period preceding SU were measured. FSF and ESF were both significantly associated with SU. FSF had a greater impact on the use of most substances than did ESF. The FSF with the strongest association with SU was lack of parental monitoring. Regarding ESF, the cumulative number of stressful external events had a higher impact on SU than previous physical or sexual assault by a stranger. In contrast, physical or sexual assault by a family member was not found to be associated with subsequent SU. These findings have important implications for SU prevention programmes focusing on male teenagers, as it is difficult to screen and intervene in subtle forms of maltreatment in families.