This work demonstrates that environmental conditions experienced by individuals can shape their development and affect the stability of genetic associations. The implication of this observation is that the environmental response may influence the evolution of traits in the wild. Here, we examined how the genetic architecture of a suite of sexually dimorphic traits changed as a function of environmental conditions in an unmanaged population of Soay sheep (Ovis aries) on the island of Hirta, St. Kilda, northwest Scotland. We examined the stability of phenotypic, genetic, and environmental (residual) covariance in males during the first year of life between horn length, body weight, and parasite load in environments of different quality. We then examined the same covariance structures across environments within and between the adult sexes. We found significant genotype-by-environment interactions for lamb male body weight and parasite load, leading to a change in the genetic correlation among environments. Horn length was genetically correlated with body weight in males but not females and the genetic correlation among traits within and between the sexes was dependent upon the environmental conditions experienced during adulthood. Genetic correlations were smaller in more favorable environmental conditions, suggesting that in good environments, loci are expressed that have sex-specific effects. The reduction in genetic correlation between the sexes may allow independent evolutionary trajectories for each sex. This study demonstrates that the genetic architecture of traits is not stable under temporally varying environments and highlights the fact that evolutionary processes may depend largely upon ecological conditions.