Identifying the sources of sediments is of great significance in reconstructing Holocene paleoclimate evolution in the Beibu Gulf and in understanding the characteristics of regional responses to changes in global climate. The Holocene paleoclimatic evolutionary history of the Beibu Gulf was investigated by chronological, geochemical, and mineralogical means using the sediments of Core KZK01 from the eastern part of the Beibu Gulf. The rare earth element (REE) distribution curves, (Gd/Yb)N and (La/Yb)N discriminant diagram, and (Gd/Lu)N and ∑LREE/∑HREE discriminant diagram indicated that the detrital materials in the eastern part of the Beibu Gulf primarily originated from Hainan Island and its proximal sources, with considerable contributions from Taiwan and Pearl River materials. Source analysis of clay minerals showed that Luzon Island was the main source of smectite, followed by Hainan Island. Rivers in Taiwan were the main sources of illite in the study area, followed by the Red River. The Red River was the main contributor of chlorite, followed by the Pearl River. Kaolinite mainly originated from Hainan Island and Guangxi. Coastal currents, surface currents, and warm currents were the main drivers of material transport. Paleoclimatic variations since the Holocene in the Beibu Gulf were divided into three stages: 12–9 cal kyr BP, 9–1.3 cal kyr BP, and 1.3 cal kyr BP to the present. During different stages of climatic evolution, drought was often accompanied by cold and humidity coexisted with warmth, and cold-dry-warm-humid alternation is characterized by significant phases. The illite crystallinity clearly recorded the extreme cold events, such as Bond Events (except Bond6) and the Younger Dryas, and the change trend was essentially consistent with the regional climate record, reflecting the control of global climate change on the process of land–sea interaction in the tropical region. Furthermore, it highlights the great potential of illite crystallinity as a proxy indicator for reconstructing the surface chemical weathering processes of the region.