Inbreeding depression, i.e., the reduction of health and vigour in individuals with high inbreeding coefficients, is expected to increase with environmental, social, or physiological stress. It has therefore been predicted that sexual selection and the associated stress usually lead to higher inbreeding depression in males than in females. However, sex-specific differences in life history may reverse that pattern during certain developmental stages. In some salmonids, for example, female juveniles start developing their gonads earlier than males who instead grow faster. We tested whether the sexes are differently affected by inbreeding during that time. To study the effects of inbreeding coefficients that may be typical for natural populations of brown trout (Salmo trutta), and also to control for potentially confounding maternal or paternal effects, we sampled males and females from the wild, used their gametes in a block-wise full-factorial breeding design to produce 60 full-sib families, released the offspring as yolk-sac larvae into the wild, sampled them 6 months later, identified their genetic sex, and used microsatellites to assign them to their parents. We used whole-genome resequencing to calculate the kinship coefficients for each breeding pair and hence the expected average inbreeding coefficient per family. Juvenile growth could be predicted from these expected inbreeding coefficients and the genetic sex: Females reached lower body sizes with increasing inbreeding coefficient, while no such link could be found in males. This sex-specific inbreeding depression led to the overall pattern that females were on average smaller than males by the end of their first summer.