Many organism traits vary along environmental gradients. Common garden experiments provide powerful means to disentangle the role of intrinsic factors, such as genetic or maternal effects, from extrinsic environmental factors in shaping phenotypic variation. Here, we investigate body size and lipid content variation in workers of the socially polymorphic ant Formica selysi along several independent elevation gradients in Switzerland. We compare field-collected workers and workers sampled as eggs from the same colonies but reared in common laboratory conditions. Overall, field-collected workers from high elevation are larger than those from low elevation, but the trend varies substantially among valleys. The same pattern is recovered when the eggs are reared in a common garden, which indicates that body size variation along elevation gradients and valleys is partly explained by genetic or maternal effects. However, both body size and lipid content exhibit significantly greater variation in field-collected workers than in laboratory-reared workers. Hence, much of the phenotypic variation results from a plastic response to the environment, rather than from genetic differences. Eggs from different elevations also show no significant difference in development time in the common garden. Overall, selection on individual worker phenotypes is unlikely to drive the altitudinal distribution of single- and multiple-queen colonies in this system, as phenotypic variation tends to be plastic and can be decoupled from social structure. This study provides insights into the interplay between individual phenotypic variation and social organization and how the two jointly respond to differing environmental conditions.