While social withdrawal is a normal defense mechanism displayed by infants to regulate interactions, it can negatively impact infant development when it becomes chronic, leading to delays in motor, cognitive, and communication difficulties in later development. Infant withdrawal was associated with low levels of parental sensitivity (i.e., the capacity of a caregiver to perceive the child signals and to respond to them accurately with an appropriate timing during interactions) in mothers. Few studies have yet been conducted in fathers and even fewer have investigated these questions in both parent-infant dyads within families, so that the joint effects of maternal and paternal sensitivity on infant social withdrawal remain unknown. We investigated within- and between-dyad associations between parental sensitivity and infant withdrawal during interactions with both parents in a sample of biparental families (n = 61) and their 3-month old infant. Results showed that higher paternal sensitivity was associated with lower infant withdrawal during father-infant interactions. The same effect was weaker in mothers and only significant when the effect of paternal sensitivity on infant withdrawal during mother-infant interaction was not taken into account. These results offer new insights about the reciprocal influences between the mother-infant and the father-infant relationships.