Animals coordinate their behavior with each other during both cooperative and agonistic social interactions. Such coordination often adopts the form of ″turn taking″, in which the interactive partners alternate the performance of a behavior. Apart from acoustic communication, how turn taking between animals is coordinated is not well understood. Furthermore, the neural substrates that regulate persistence in engaging in social interactions are poorly studied. Here, we use Siamese fighting fish (Betta splendens), to study visually-driven turn-taking aggressive behavior. Using encounters with conspecifics and with animations, we characterize the dynamic visual features of an opponent and the behavioral sequences that drive turn taking. Through a brain-wide screen of neuronal activity during coordinated and persistent aggressive behavior, followed by targeted brain lesions, we find that the caudal portion of the dorsomedial telencephalon, an amygdala-like region, promotes persistent participation in aggressive interactions, yet is not necessary for coordination. Our work highlights how dynamic visual cues shape the rhythm of social interactions at multiple timescales, and points to the pallial amygdala as a region controlling engagement in such interactions. These results suggest an evolutionarily conserved role of the vertebrate pallial amygdala in regulating the persistence of emotional states.