Asian American panethnicity was conceptualized to unify ethnic groups and represent their sociopolitical interests. Increasingly however, scholars have questioned whether panethnicity accurately reflects the diversity of different ethnic groups’ experiences and identities. In mainstream culture, “Asian American” has become synonymous with East Asian Americans and stereotypes—albeit biased ones—of their affluence, thus erasing the realities of working-class, South, and Southeast Asian Americans (SEAAs). I focus on the last group and join other scholars in emphasizing how ethnic groups’ unique historical relationships with the United States differentially impact their racial identities and attachments to panethnicity. Using 62 interviews with Southeast Asian refugees and service providers in North Carolina, I explore how a term I call “quiet neglect”—the U.S. institutionalized silence around the Vietnam and Secret Wars that has led to an erasure of SEAAs’ needs—shape their connections to Asian American panethnicity and decision to align with alternative identities. At stake in this study is our capacity to recognize individuals’ agency to challenge racial boundaries and assert identities that they find meaningful. In addition, I examine how SEAAs situate themselves within our broader racial structure and harness their identities to connect with other people of color.