Background: The ocular involvement of neuroendocrine neoplasms (NENs) is uncommon and mainly represented by metastases from gastrointestinal and lung neuroendocrine tumors. Primary orbital NENs are even less common and their diagnostic and therapeutic management is a challenge. Methods: A systematic review of the literature was conducted from 1966 to September 2023 on PubMed to identify articles on orbital NENs and to summarize their clinical–pathological features, diagnosis and therapeutic management. Furthermore, we presented a case of a locally advanced retro-orbital primary neuroendocrine tumor that was referred to the certified Center of Excellence of Sant’Andrea Hospital, La Sapienza University of Rome, Italy. Results: The final analysis included 63 records on orbital NENs and 11 records focused on primary orbital NENs. The localization was mostly unilateral and in the right orbit; proptosis or exophthalmos represented the initial symptoms. The diagnostic work-up and therapeutic management was discussed and a diagnostic algorithm for the suspicion of primary orbital NENs was proposed. Conclusions: A multidisciplinary approach is required for the management of primary orbital NENs, emphasizing the importance of early referral to dedicated centers for prompt differential diagnosis, tailored treatment, and an improved quality of life and survival.