In the e-commerce environment, conversations between customers and businesses contain a multitude of useful information about customer sentiment. By mining that information, customer sentiment can be validly identified, which is helpful in accurately identifying customer needs and improving customer satisfaction. Contextual semantics information and inter-speaker affective interaction information are two key factors for identifying customers’ sentiments in conversation. Unfortunately, none of the existing approaches consider the two factors simultaneously. In this paper, we propose a conversational sentiment analysis method based on contextual semantic and affective interaction information. The proposed approach uses different bidirectional gated recurrent unit (BiGRU) combined with attention mechanisms to encode the contextual semantic information of different types of conversational texts. For modeling affective interactions, we use directed graph structures to portray the affective interactions between speakers and encode them with affective interaction features using graph convolutional neural networks (GCN). Finally, the two features are afused to recognize customer sentiment. The experimental results on the JDDC dataset show that our model can more accurately recognize customer sentiment than other baseline models in customer service conversation.