This study aims to examine the conflict between loyal customers and non-loyal customers in the luxury hotel environment. This research mainly applied the concept of brand tourism effect and employed a 2 (two types of non-loyal customers: brand immigrants and brand tourists) x 2 (areas divided by psychological ownership: lounge vs. restaurant) mixed experimental design on the dependent variables (i.e., emotions, infringement, attitude, brand image, and switching intention) and 934 participants were recruited from nationwide online panels. To investigate loyal customers' different brand perceptions from non-loyal customers, MANOVA was performed and to examine the relationship among infringement, attitude, brand image, and switching intention, PLSSEM was conducted. The result revealed that loyal customers negatively perceive brand immigrants and positively perceive brand tourists. Loyal customers perceive higher levels of infringement and switching intention when they view brand immigrants, while loyal customers perceive positive attitude and brand image when they view brand tourists. Psychological ownership, which divided the background into lounge and restaurant, is also found to play a moderating role in loyal customers' perceptions of brand tourists and brand immigrants on infringement, attitude, and switching intention. In addition, for brand immigrants, infringement is positively related to switching intention, and attitude and brand image are negatively linked with switching intention, while for brand tourists, infringement is the only variable that has a positive influence on switching intention. In sum, the current study demonstrated that loyal customers perceive brand differently depending on the types of customers (i.e., brand tourists and brand immigrants) and two types of backgrounds (i.e., lounge and restaurant). Finally, the theoretical and empirical contributions of the present study are discussed.