This paper characterises families where the father is not living (or not living permanently) with the child from around birth, and identifies the drivers of the evolution of father contact over the first year of life across different types of household. We use a recent, nationally representative cohort of children born in France in 2011, Elfe (the Etude longitudinale française depuis l’enfance), and latent clustering techniques to identify different groups of households characterised by non-residential fatherhood. We show that non-residential fatherhood from around birth is not a marginal phenomenon in France, and it corresponds to a heterogeneity of situations, describing both advantaged and low involvement fathers, as well less disadvantaged but involved groups. Over the first year of life, most non-resident fathers managed to keep in contact with their child, including relatively disadvantaged groups such as migrant and young parents, although groups characterised by low father involvement shortly after birth lost contact. On the other hand, among a group of very involved non-resident fathers who were in a relationship with the mother, we observed high levels of contact and indeed co-residence when the child was one year of age. A number of channels emerged to explain the correlations between our latent groups and father contact at one year: notably, father engagement around birth, especially whether the father formally recognised the child. Trajectories of father–child involvement and of parental relationships are therefore at least as important as socio-economic conditions to understand future father contact.