Axelrod's model for the dissemination of culture combines two key ingredients of social dynamics: social influence, through which people become more similar when they interact, and homophily, which is the tendency of individuals to interact preferentially with similar others. In that agentbased model, the agents are fixed to the nodes of a network and are allowed to interact with a predetermined set of peers only, resulting in the frustration of the agents that end up at the boundaries of the cultural domains. Here we modify Axelrod's model by allowing the possibility that the agents move away from their cultural opposites and stay put when near their cultural likes. Hence the social appeal or attractivity between any two agents is proportional to their cultural similarity and determines the odds that those agents will move apart or stay put. We find that the homophilydriven mobility fragments severely the influence network for low initial cultural diversity, resulting in a network composed of a macroscopic number of microscopic components in the thermodynamic limit. For high initial cultural diversity, we find that a macroscopic component coexists with the microscopic ones. The transition between these two fragmentation regimes changes from continuous to discontinuous as the step size increases and disappears altogether at a critical end point, so that for large step sizes the influence network is severely fragmented. Regardless of the fragmentation regime, we find that the absorbing configurations are always multicultural for nonzero step sizes.