It has long been predicted that the degree to which language is lateralized to the left or right hemisphere might be reflected in the underlying brain anatomy. We investigated this relationship on a voxel-by-voxel basis across the whole brain using structural and functional magnetic resonance images from 86 healthy participants. Structural images were converted to gray matter probability images, and language activation was assessed during naming and semantic decision. All images were spatially normalized to the same symmetrical template, and lateralization images were generated by subtracting right from left hemisphere signal at each voxel. We show that the degree to which language was left or right lateralized was positively correlated with the degree to which gray matter density was lateralized. Post hoc analyses revealed a general relationship between gray matter probability and blood oxygenation level-dependent signal. This is the first demonstration that structural brain scans can be used to predict language lateralization on a voxel-by-voxel basis in the normal healthy brain.