We provide an agent-based model to explain the emergence of collective opinions not based on feedback between different opinions, but based on emotional interactions between agents. The driving variable is the emotional state of agents, characterized by their valence, quantifying the emotion from unpleasant to pleasant, and their arousal, quantifying the degree of activity associated with the emotion. Both determine their emotional expression, from which collective emotional information is generated. This information feeds back on the dynamics of emotional states and individual opinions in a nonlinear manner. We derive the critical conditions for emotional interactions to obtain either consensus or polarization of opinions. Stochastic agent-based simulations and formal analyses of the model explain our results. Possible ways to validate the model are discussed.