Extreme prematurity and pregnancy conditions leading to intrauterine growth restriction (IUGR) affect thousands of newborns every year and increase their risk for poor higher order cognitive and social skills at school age. However, little is known about the brain structural basis of these disabilities. To compare the structural integrity of neural circuits between prematurely born controls and children born extreme preterm (EP) or with IUGR at school age, long-ranging and short-ranging connections were noninvasively mapped across cortical hemispheres by connection matrices derived from diffusion tensor tractography. Brain connectivity was modeled along fiber bundles connecting 83 brain regions by a weighted characterization of structural connectivity (SC). EP and IUGR subjects, when compared with controls, had decreased fractional anisotropy-weighted SC (FAw-SC) of cortico-basal ganglia-thalamo-cortical loop connections while cortico-cortical association connections showed both decreased and increased FAw-SC. FAw-SC strength of these connections was associated with poorer socio-cognitive performance in both EP and IUGR children.