The color word Stroop effect in bilinguals is commonly half the magnitude when the written and naming languages are different (between) than when they are the same (within). This between-within language Stroop difference (BWLS) is likened to a response set effect, with greater response conflict for response relevant than irrelevant words. The nature of the BWLS was examined using a bilingual Stroop task. In a given block (Experiment 1), color congruent and incongruent words appeared in the naming language or not (single), or randomly in both languages (mixed). The BWLS effect was present for both balanced and unbalanced bilinguals, but only partially supported a response set explanation. As expected, color incongruent trials during single language blocks, lead to slower response times within than between languages. However, color congruent trials during mixed language blocks led to slower times between than within languages, indicating that response-irrelevant stimuli interfered with processing. In Experiment 2, to investigate the neural timing of the BWLS effect, event related potentials were recorded while balanced bilinguals named silently within and between languages. Replicating monolingual findings, an N450 effect was observed with larger negative amplitude for color incongruent than congruent trials (350–550 ms post-stimulus onset). This effect was equivalent within and between languages, indicating that color words from both languages created response conflict, contrary to a strict response set effect. A sustained negativity (SN) followed with larger amplitude for color incongruent than congruent trials, resolving earlier for between than within language Stroop. This effect shared timing (550–700 ms), but not morphology or scalp distribution with the commonly reported sustained potential. Finally, larger negative amplitude (200–350 ms) was observed between than within languages independent of color congruence. This negativity, likened to a no-go N2, may reflect processes of inhibitory control that facilitate the resolution of conflict at the SN, while the N450 reflects parallel processing of distracter words, independent of response set (or language). In sum, the BWLS reflects brain activity over time with contributions from language and color conflict at different points.