Neuroendocrine neoplasms (NENs) of the major and minor ampulla are rare diseases with clinico-pathologic features distinct from non-ampullary-duodenal NENs. However, they have been often combined and the knowledge on prognostic factors specific to ampullary NENs (Amp-NENs) is limited. The aim of this study was to identify factors associated with metastatic potential and patient prognosis in Amp-NENs. We clinically and histologically investigated an international series of 119 Amp-NENs, comprising 93 ampullary neuroendocrine tumors (Amp-NETs) and 26 neuroendocrine carcinomas (Amp-NECs). Somatostatin-producing tubulo-acinar NET represented the predominant Amp-NET histologic subtype (58 cases, 62%, 12 associated with type 1 neurofibromatosis). Compared to Amp-NETs, Amp-NECs arose in significantly older patients and showed a larger tumor size, a more frequent small vessel invasion, a deeper level of invasion and a higher rate of distant metastasis, and, importantly, a tremendously worse disease-specific patient survival. In Amp-NETs, the WHO grade proved to be a strong predictor of disease-specific survival (hazard ratio: 12.61, p < 0.001 for G2 vs G1), as well as patient age at diagnosis > 60 years, small vessel invasion, pancreatic invasion, and distant metastasis at diagnosis. Although nodal metastatic disease was not associated with survival by itself, patients with > 3 metastatic lymph nodes showed a worse outcome in comparison with the remaining Amp-NET cases with lymphadenectomy. Tumor epicenter in the major ampulla, small vessel invasion, and tumor size > 16 mm were independent predictors of nodal metastases in Amp-NETs. In conclusion, we identified prognostic factors, which may eventually help guide treatment decisions in Amp-NENs.