The phenotypic consequences of a given mutation can vary across individuals. This so-called background effect is widely observed, from mutant fitness of loss-of-function variants in model organisms to variable disease penetrance and expressivity in humans; however, the underlying genetic basis often remains unclear. Taking insights gained from recent large-scale surveys of genetic interaction and suppression analyses in yeast, we propose that the genetic network context for a given mutation may shape its propensity of exhibiting background-dependent phenotypes. We argue that further efforts in systematically mapping the genetic interaction networks beyond yeast will provide not only key insights into the functional properties of genes, but also a better understanding of the background effects and the (un)predictability of traits in a broader context.