Ellagic acid, known for its various biological activities, is widely used. Ellagic acid from pomegranate peels is safe for consumption, while that from gallnuts is only suitable for external use. However, there is currently no effective method to confirm the source of ellagic acid. Therefore, this study establishes an analysis method using ultra-high-performance liquid chromatography–electrospray ionization–high-resolution mass spectrometry (UHPLC-ESI-HR-MS) to identify the components of crude ellagic acid extracts from pomegranate peels and gallnuts. The analysis revealed that there was a mix of components in the crude extracts, such as ellagic acid, palmitic acid, oleic acid, stearic acid, and 9(10)-EpODE. Furthermore, it could be observed that ellagic acid extracted from gallnuts contained toxic substances such as anacardic acid and ginkgolic acid (15:1). These components could be used to effectively distinguish the origin of ellagic acid from pomegranate peels or gallnuts. Additionally, a rapid quantitative analysis method using UHPLC-ESI-MS with multiple reaction monitoring (MRM) mode was developed for the quality control of ellagic acid products, by quantifying anacardic acid and ginkgolic acid (15:1). It was found that one of three ellagic acid health care products contained ginkgolic acid (C15:1) and anacardic acid at more than 1 ppm.