The integration of automation and shared mobility services would significantly affect transportation demand, especially mode choice. However, little is known about how attitudes, travel attributes, and demographic factors affect the modal shift to shared autonomous vehicles (SAVs). A stated preference survey was designed to determine the preferences of car and transit users in relation to a modal shift to SAVs. The binary logit models’ results revealed distinct behavior patterns and systematic heterogeneity among transit and private car users based on a representative sample of 607 individuals in 2021. The shifting behavior of both users is positively affected by attitudinal factors, including consumer innovativeness, perceived usefulness, sharing intention, and ecological awareness, while negatively affected by privacy concerns. In terms of travel-related attributes of SAVs, car users are eight times more sensitive to waiting times compared to transit users, who are three times more concerned with travel costs. Further, privacy concerns, the number of passengers sharing a trip, and the ratio of waiting time to travel time of SAVs were the major barriers to shifting the likelihood of car users’ behavior. In light of these findings, based on the likely effects of SAVs on shifting behavior, a number of practical implications are suggested for more effective policy making.