Decades of research suggest that encoding information with respect to the self improves memory (self-reference effect, SRE) for items (item SRE). The current study focused on how processing information in reference to the self affects source memory for whether an item was self-referentially processed (a source SRE). Participants self-referentially or non-self-referentially encoded words (Experiment 1) or pictures (Experiment 2) that varied in valence (positive, negative, neutral). Relative to non-self-referential processing, self-referential processing enhanced item recognition for all stimulus types (an item SRE), but it only enhanced source memory for positive words (a source SRE). In fact, source memory for negative and neutral pictures was worse for items processed self-referentially than non-self-referentially. Together, the results suggest that item SRE and source SRE (e.g., remembering an item was encoded self-referentially) are not necessarily the same across stimulus types (e.g., words, pictures; positive, negative). While an item SRE may depend on the overall likelihood the item generates any association, the enhancing effects of self-referential processing on source memory for self-referential encoding may depend on how embedded a stimulus becomes in one’s self-schema, and that depends, in part, on the stimulus’ valence and format. Self-relevance ratings during encoding provide converging evidence for this interpretation.