Despite prognostic grading and staging systems, it is a challenge to predict outcomes for patients with pancreatic neuroendocrine tumors (PanNETs). Sequencing studies of PanNETs have identified alterations in death domain-associated protein (DAXX) and ATRX chromatin remodeler (ATRX). In tumors, mutations in DAXX or ATRX and corresponding loss of protein expression correlate with shorter times of disease-free survival and disease-specific survival of patients. However, DAXX or ATRX proteins were lost in only 50% of distant metastases analyzed. We performed whole-exome sequencing analyses of 20 distant metastases from 20 patients with a single non-syndrome, non-functional PanNET. We found distant metastases contained alterations in MEN1 (n=8), ATRX (n=5), DAXX (n=5), TSC2 (n=3), and DEPDC5 (n=3). We found copy number loss of CDKN2A in 15 metastases (75%) and alterations in genes that regulate chromatin remodeling including SETD2 (n=4), ARID1A (n=2), CHD8 (n=2), and DNMT1 (n=2). In a separate analysis of 347 primary PanNETs, we found loss or deletion of DAXX and ATRX, disruption of SETD2 function (based on loss of H3K36me3), loss of ARID1A expression or deletions in CDKN2A in 81% of primary PanNETs with distant metastases. Among patients with loss or deletion of at least 1 of these proteins or genes, 39% survived disease free for 5 years and 44% had disease-specific survival times of 10 years. Among patients without any of these alterations, 98% survived disease free for 5 years and 95% had disease-specific survival times of 10 years. Therefore, primary PanNETs with loss of DAXX, ATRX, H3K36me3, ARID1A, and/or CDKN2A associate with shorter survival times of patients. Our findings indicate that alterations in chromatin remodeling genes and CDKN2A contribute to metastasis of PanNETs.