Adaptive social behavior and mental well-being depend on not only recognizing emotional expressions but also, inferring the absence of emotion. While the neurobiology underwriting the perception of emotions is well studied, the mechanisms for detecting a lack of emotional content in social signals remain largely unknown. Here, using cutting-edge analyses of effective brain connectivity, we uncover the brain networks differentiating neutral and emotional body language. The data indicate greater activation of the right amygdala and midline cerebellar vermis to nonemotional as opposed to emotional body language. Most important, the effective connectivity between the amygdala and insula predicts people's ability to recognize the absence of emotion. These conclusions extend substantially current concepts of emotion perception by suggesting engagement of limbic effective connectivity in recognizing the lack of emotion in body language reading. Furthermore, the outcome may advance the understanding of overly emotional interpretation of social signals in depression or schizophrenia by providing the missing link between body language reading and limbic pathways. The study thus opens an avenue for multidisciplinary research on social cognition and the underlying cerebrocerebellar networks, ranging from animal models to patients with neuropsychiatric conditions.