The distribution of ice‐rafted detritus (IRD) is studied in three cores from the western Svalbard slope (1130–1880 m water depth, 76–78°N) covering the period 74–0 ka. The aim was to provide new insight into the dynamics of the Svalbard–Barents Sea Ice Sheet during Marine Isotope Stages (MIS) 4–1 to get a better understanding of ice‐sheet interactions with changes in ocean circulation and climate on orbital and millennial (Dansgaard–Oeschger events of stadial–interstadial) time scales. The results show that concentration, flux, composition and grain‐size of IRD vary with climate and ocean temperature on both orbital and millennial time scales. The IRD consists mainly of fragments of siltstones and mono‐crystalline transparent quartz (referred to as ‘quartz’). IRD dominated by siltstones has a local Svalbard–Barents Sea source, while IRD dominated by quartz is from distant sources. Local siltstone‐rich IRD predominates in warmer climatic phases (interstadials), while the proportion of allochthonous quartz‐rich IRD increases in cold phases (glacials and stadials/Heinrich events). During the Last Glacial Maximum and early deglaciation at 24–16.1 ka, the quartz content reached up to >90%. In warm climate, local iceberg calving apparently increased and the warmer ocean surface caused faster melting. During the glacial maxima (MIS 4 and MIS 2) and during cold stadials and Heinrich events, the local ice‐sheets must have been relatively stable with low ablation. During ice retreat phases of the MIS 4/3 and MIS 2/1 transitions, maxima in IRD deposition were dominated by local coarse‐grained IRD. These maxima correlate with episodes of climate warming, indicating a rapid, stepwise retreat of the Svalbard–Barents Sea Ice Sheet in phase with millennial‐scale climate oscillations.