A considerable amount of research has been devoted to understanding individual differences in lexical learning, however, the majority of this research has been conducted with spoken languages rather than signed languages and thus we know very little about the cognitive processes involved in sign learning or the extent to which lexical learning processes are specific to word learning. The present study was conducted to address this gap. Two-hundred thirty-six non-signing adults completed 25 tasks assessing word learning and sign learning (via associative learning paradigms) as well as modality-specific phonological short-term memory, working memory capacity, crystallized intelligence, and fluid intelligence. Latent variable analyses indicated that, when other variables were held constant, fluid intelligence was predictive of both word and sign learning, however, modality-specific phonological short-term memory factors were only predictive of lexical learning within modality-none of the other variables made significant independent contributions. It was further observed that sign and word learning were strongly correlated. Exploratory analyses revealed that all lexical learning tasks loaded onto a general factor, however, sign learning tasks loaded onto an additional specific factor. As such, this study provides insight into the cognitive components that are common to associative L2 lexical learning regardless of language modality and those that are unique to either signed or spoken languages. Results are further discussed in light of established and more recent theories of intelligence, short-term memory, and working memory.