Adverse conditions during fetal life have been associated to both structural and functional changes in neurodevelopment from the neonatal period to adolescence. In this study, connectomics was used to assess the evolution of brain networks from infancy to early adolescence. Brain network reorganization over time in subjects who had suffered adverse perinatal conditions is characterized and related to neurodevelopment and cognition. Three cohorts of prematurely born infants and children (between 28 and 35 weeks of gestational age), including individuals with a birth weight appropriated for gestational age and with intrauterine growth restriction (IUGR), were evaluated at 1, 6, and 10 years of age, respectively. A common developmental trajectory of brain networks was identified in both control and IUGR groups: network efficiencies of the fractional anisotropy (FA)-weighted and normalized connectomes increase with age, which can be related to maturation and myelination of fiber connections while the number of connections decreases, which can be associated to an axonal pruning process and reorganization. Comparing subjects with or without IUGR, a similar pattern of network differences between groups was observed in the three developmental stages, mainly characterized by IUGR group having reduced brain network efficiencies in binary and FA-weighted connectomes and increased efficiencies in the connectome normalized by its total connection strength (FA). Associations between brain networks and neurobehavioral impairments were also evaluated showing a relationship between different network metrics and specific social cognition-related scores, as well as a higher risk of inattention/hyperactivity and/or executive functional disorders in IUGR children.