Introduction: The prognosis and impact of different prognostic factors in pancreatic neuroendocrine neoplasms (pNEN) remain controversial. Aim: To investigate prognostic factors for recurrence-free survival and disease-specific survival in patients with pNEN, divided into three groups: patients undergoing surveillance (tumor size < 2 cm, group 1), patients followed after curative-intended surgery (group 2), and patients with unresectable disease or residual tumors after resection (group 3). Method: A single-center retrospective study including consecutive patients over a 20-year period. Multivariate Cox regression analyses were performed to identify risk factors. Results: 413 patients were included, with a mean (SD) age of 62 ± 14 years. In group 1 (n = 51), median (IQR) follow-up was 29 (21–34) months, and tumor size was 1.0 (0.8–1.4) cm. One progressed and had a tumor resection. In group 2 (n = 165), follow-up 59 (31–102) months, median tumor size 2 (1.2–3.4) cm, median Ki-67 index 5 (3–10)%, the 5-year recurrence rate was 21%. Tumor size (p < 0.001), Ki-67 index (p = 0.02), and location in the pancreatic head (p < 0.001) were independent risk factors. In group 3 (n = 197), follow-up 19 (6–46) months, median tumor size 4.2 (2.6–7.0) cm, Ki-67 index 17 (9–64)%, the median disease-specific survival was 22 (6–75) months—99 in NET G1; 54 in NET G2; 14 in NET G3; and 6 months in neuroendocrine carcinomas (NEC). Age (p = 0.029), plasma chromogranin A (p = 0.014), and proliferation, expressed by grade (p = 0.001) and Ki-67 index (p < 0.001), were risk factors. Conclusion: Growth in pNET < 2 cm requiring surgery was observed in 1/51. Tumor size, Ki-67 index, and location in the head were prognostic factors for disease recurrence, while age, plasma chromogranin A, and proliferation predicted mortality in patients with unresectable disease or residual tumors after resection.