Packaging design has a non-negligible impact on consumers’ emotional feedback and purchase intention, and color, as one of the most important parts of packaging design, will also have an impact on consumer psychology. Based on the psychology of consumers in color consumption in packaging design, the article designs and constructs a multimodal learning-based model for predicting consumer emotions. The model is divided into a modality-specific learning module, a cross-modal fusion module, and an emotion prediction module to recognize and predict consumer emotions. At the same time, K-means, C-means, and ISO-DATA are introduced to extract features from colors in packaging design. The emotional imagery and positive-negative emotional bias of colors in this paper’s model are analyzed, and the accuracy of emotional prediction and training time of this paper’s model are examined. Gold, pink, green, orange, white, and yellow all have an emotional bias score greater than 2.5 and are classified as “positive” emotional colors. Purple, red, brown, blue, black, and grey were identified as “negative” emotion colors because their emotion bias values were less than 2.5. This paper’s model has the most accurate prediction accuracy and the shortest training time among the bimodal, trimodal, and quaternary sentiment analysis experiments. The model’s prediction accuracy increases as the number of modalities increases, but the training time also increases.