For the histopathological work-up of resected neuroendocrine tumors of the small intestine (siNET), the determination of lymphatic (LI), microvascular (VI) and perineural (PnI) invasion is recommended. Their association with poorer prognosis has already been demonstrated in many tumor entities. However, the influence of LI, VI and PnI in siNET has not been sufficiently described yet. A retrospective analysis of all patients treated for siNET at the ENETS Center of Excellence Charité–Universitätsmedizin Berlin, from 2010 to 2020 was performed (n = 510). Patients who did not undergo primary resection or had G3 tumors were excluded. In the entire cohort (n = 161), patients with LI, VI and PnI status had more distant metastases (48.0% vs. 71.4%, p = 0.005; 47.1% vs. 84.4%, p < 0.001; 34.2% vs. 84.7%, p < 0.001) and had lower rates of curative surgery (58.0% vs. 21.0%, p < 0.001; 48.3% vs. 16.7%, p < 0.001; 68.4% vs. 14.3%, p < 0.001). Progression-free survival was significantly reduced in patients with LI, VI or PnI compared to patients without. This was also demonstrated in patients who underwent curative surgery. Lymphatic, vascular and perineural invasion were associated with disease progression and recurrence in patients with siNET, and these should therefore be included in postoperative treatment considerations.