Many studies of speech accommodation focus on native speakers with different dialects, whereas only a limited number of studies work on L2 speakers’ accommodation and discuss theories for second language (L2) accommodation. This paper aimed to fill the theoretical gap by integrating the revised speech learning model (SLM) with the exemplar-based models for L2 speech accommodation. A total of 19 Cantonese–English bilingual speakers completed map tasks with English speakers of Received Pronunciation and General American English in two separate experiments. Their pronunciations of THOUGHT and PATH vowels, and fricatives [z] and [θ] were examined before, during, and after the map tasks. The role of phonetic dissimilarity in L2 accommodation and L2 category formation in the revised SLM (SLM-r) were tested. First, the results suggested that global phonetic dissimilarity cannot predict Hong Kong English (HKE) speakers’ accommodation patterns. Instead, the segment-specific phonetic dissimilarity between participants and interlocutors was found to be positively correlated with the participants’ degree of accommodation. In addition, HKE speakers who did not form a new L2 category of [z] were found to significantly accommodate toward their interlocutor, suggesting that L2 accommodation might not be constrained by phonological category. An integrated exemplar model for L2 accommodation is proposed to explain these findings.