We used event-related potentials to examine the precise moment at which talker-specific information comes into play during spoken word recognition. Dichotic repetition priming was examined with primes presented in the left unattended ear and targets presented in the right attended ear, and we manipulated both word frequency and talker identity. A clear repetition priming effect was observed in an early time-window spanning 100–200 ms post-target onset, and the effect continued after target word offset in a time-window between 650 and 800 ms. Crucially, we observed that talker change caused a diminution in repetition priming only in the N400 time-window, and only for low frequency words but not for high frequency words. Together, our findings suggest that spoken word recognition relies primarily on abstract representations, and that talker-specific information mainly affects later stages of this process, namely lexical selection.