In recent years, applied linguists have paid more attention to the contributory roles of vocabulary knowledge in language subskills. However, little evidence has been collected to shed light upon the cross‐modality relationship between the two modalities of receptive word knowledge, namely phonological and orthographic vocabulary, and receptive language skills, namely reading and listening comprehension. Moreover, the majority of studies conducted on the topic only examined the bivariate correlations between the modalities and did not take the shared variance of the constructs into account, making their findings superficial. The present study analyzed a dataset which reported the performance of 234 students on two validated tests of receptive phonological and orthographic vocabulary knowledge and two standardized tests of listening and reading proficiency. Multi‐group, covariance‐based structural equation modeling was applied. Results from the analyses showed that both aural and orthographic knowledge of words were of equal importance to learners’ receptive vocabulary knowledge. While whole‐group results revealed that vocabulary knowledge contributed equally to learners’ reading and listening proficiency, multi‐group analyses indicated noticeable differences in the contributory role of receptive word knowledge in comprehension between learners of different vocabulary size groups. The role of vocabulary size in the relationship between vocabulary knowledge and comprehension was discussed and practical implications for vocabulary assessment were provided.