Behavioral studies have shown that matching individual faces across depth rotation is easier and faster for familiar than unfamiliar faces. Here we used event-related potentials (ERPs) to clarify the locus of this behavioral facilitation, that is whether it reflects changes at the level of perceptual face encoding, or rather at later stages of processing. We used an identity adaptation paradigm in ERPs, during which a first (adapting) face (~3000 ms) rotated 30° in depth was followed by a second full front face (200 ms) which was either the same or a different identity as the first face. For unfamiliar faces, the early face-sensitive N170 component was reduced for immediately repeated as compared to different unfamiliar faces in the right hemisphere only. However, for personally familiar faces, the effect was absent at right hemisphere electrode sites and appeared instead over the left hemisphere at the same latency. Later effects of face identity adaptation were also present on the scalp, but from about 300 to 400 ms over fronto-central regions, and slightly later on occipito-temporal regions, there was a strong adaptation effect only for familiar faces. These observations suggest that the perceptual encoding of familiar and unfamiliar faces may be of different nature, as indicated by early (N170) hemispheric differences for identity adaptation effects depending on long-term familiarity. However, the behavioral advantage provided by familiarity to match faces across viewpoints might rather be related to processes that are closer in time to the behavioral response, such as semantic associations between the faces to match.