In second language research, the concept of cross-linguistic influence or transfer has frequently been used to describe the interaction between the first language (L1) and second language (L2) in the L2 acquisition process. However, less is known about the L2 acquisition of a sign language in general and specifically the differences in the acquisition process of L2M2 learners (learners learning a sign language for the first time) and L2M1 learners (signers learning another sign language) from a multimodal perspective. Our study explores the influence of modality knowledge on learning Swedish Sign Language through a descriptive analysis of the sign lexicon in narratives produced by L2M1 and L2M2 learners, respectively. A descriptive mixed-methods framework was used to analyze narratives of adult L2M1 (n = 9) and L2M2 learners (n = 15), with a focus on sign lexicon, i.e., use and distribution of the sign types such as lexical signs, depicting signs (classifier predicates), fingerspelling, pointing, and gestures. The number and distribution of the signs are later compared between the groups. In addition, a comparison with a control group consisting of L1 signers (n = 9) is provided. The results suggest that L2M2 learners exhibit cross-modal cross-linguistic transfer from Swedish (through higher usage of lexical signs and fingerspelling). L2M1 learners exhibits same-modal cross-linguistic transfer from L1 sign languages (through higher usage of depicting signs and use of signs from L1 sign language and international signs). The study suggests that it is harder for L2M2 learners to acquire the modality-specific lexicon, despite possible underlying gestural knowledge. Furthermore, the study suggests that L2M1 learners’ access to modality-specific knowledge, overlapping access to gestural knowledge and iconicity, facilitates faster L2 lexical acquisition, which is discussed from the perspective of linguistic relativity (including modality) and its role in sign L2 acquisition.