Both the concept of a “native speaker/signer” and the term itself have been critiqued and problematized across fields such as linguistic anthropology, second language acquisition, and English language teaching. However, within less peripheralized disciplines of linguistics, this term is often used uncritically, with some disciplines centering the behavior and knowledge of “native speakers” as the main object of inquiry. We argue that the term “native speaker/signer” is a colonial analytic that reifies the connection between named languages and nation-states, essentializes language in identity formation, and produces harmful consequences. Given that previous work has problematized the notion of a “native speaker” and enumerated its pitfalls (e.g., Holliday 2006, Love & Ansaldo 2010), we seek to understand how the term became and continues to be naturalized and why it is still used today; a historical approach is crucial to understanding both the colonial roots and the transformation of the term. We track the evolution and proliferation of “native speaker” from its inception, trace its role in linguistic theory, and note how it has been critiqued in linguistic anthropology. Looking forward, we emphatically do not propose substituting the term “native speaker” with other discrete labels that have similar colonial ideological underpinnings such as “L1 speaker." Rather, we argue that linguists must interrogate the role of this concept and its associated theoretical baggage. We anticipate that moving away from this term and the ideas surrounding it will (a) lead to more specified hypotheses about how particular types of language experience might affect linguistic knowledge/behavior, (b) encourage analyses which take into account the diverse linguistic repertoires held by all language users, (c) force our theoretical approaches to center non-normative language use, and (d) be one step towards a more inclusive discipline.