The effect of metals on the functioning of the human eye is multifactorial and includes enzyme activity modulation, trace metal metabolic pathways changes, and cytotoxic activity. Functional dysfunctions appear mostly as a result of the accumulation of toxic xenobiotic metals or disturbances of micronutrients’ homeostasis. So far, the affinity of selected metals to eye tissues, i.e., the cornea, choroid, lens, and anterior chamber fluid, has been most studied. However, it is known that many eye symptoms are related to damage to the optic nerve. In order to fill this gap, the aim of the study is to perform a multi-element analysis of tissue collected postmortem from optic chiasm and optic nerves. A total of 178 samples from 107 subjects were tested. The concentrations of 51 elements were quantified by inductively coupled plasma mass spectrometry (ICP-MS) after the wet-mineralization step. In terms of elemental composition, the optic chiasm is dominated by two trace elements, i.e., iron (Fe) and zinc (Zn), besides macro-elements Ca, K, Na, P, and Mg. The subjects formed a homogeneous cluster (over 70% subjects) with the highest accumulation of aluminum (Al). The remaining two departing clusters were characterized by an increased content of most of the elements, including toxic elements such as bismuth (Bi), uranium (U), lead (Pb), chromium (Cr), and cadmium (Cd). Changes in elemental composition with age were analyzed statistically for the selected groups, i.e., females, males, and subjects with alcohol use disorder (AUD) and without AUD. A tendency of women to lose Se, Cu, Zn, Fe with age was observed, and a disturbed Ca/Mg, Na/K ratio in subjects with AUD. Although the observed trends were not statistically significant, they shed new light on the risks and possible pathologies associated with metal neurotoxicity in the visual tract.