Interpersonal touch buffers against stress under challenging conditions, but this effect depends on familiarity. People benefit from receiving touch from their romantic partners, but the results are less consistent in the context of receiving touch from an opposite-gender stranger. We propose that there may be important gender differences in how people respond to touch from opposite-gender strangers. Specifically, we propose that touch from an opposite-gender stranger may only have stress-buffering effects for men, not women. Stress was induced as participants took part in an emotion recognition task in which they received false failure feedback while being touched from a romantic partner or stranger. We measured subjective and physiological markers of stress (i.e., reduced heart rate variability) throughout the experiment. Neither stranger’s nor partner’s touch had any effect on subjective or physiological markers of stress for men. Women, however, subjectively experienced a stress-buffering effect of partner and stranger touch, but showed increased physiological markers of stress when receiving touch from an opposite-gender stranger. These results highlight the importance of considering gender when investigating touch as a stress buffer.