Neuroendocrine differentiation in breast carcinomas has been a matter of discussion since it was first described almost 60 years ago. Indeed, so-called neuroendocrine neoplasms of the breast (Br-NENs) are a less well-defined group of neoplasms than analogous entities in other anatomic sites, such as the lung and the gastroenteropancreatic (GEP) tract. Pure neuroendocrine phenotype is extremely rare, whereas the expression of neuroendocrine markers in usual breast carcinomas, both of special and of non-special type, without evident neuroendocrine morphology, is more common. In this context, the diagnostic criteria and the classification scheme for Br-NENs have been continuously changing over time and real consensus on this topic is still lacking, despite the recent publication of the 5th edition of the WHO classification of breast tumors. In this review, we will recapitulate the evolution of the concept of Br-NEN; revise the available knowledge on their morphological, molecular, and clinical features; and critically discuss the current classification scheme.