Understanding the process of producing creative responses to open-ended problems solved in small groups is important for many modern domains, like health care, manufacturing, education, banking, and investment. Some of the main theoretical challenges include characterizing and measuring the dynamics of responses, relating social and individual aspects in group problem solving, incorporating soft skills (e.g., experience, social aspects, and emotions) to the theory of decision making in groups, and understanding the evolution of processes guided by soft utilities (hard-to-quantify utilities), e.g., social interactions and emotional rewards. This paper presents a novel theoretical model (TM) that describes the process of solving open-ended problems in small groups. It mathematically presents the connection between group member characteristics, interactions in a group, group knowledge evolution, and overall novelty of the responses created by a group as a whole. Each member is modeled as an agent with local knowledge, a way of interpreting the knowledge, resources, social skills, and emotional levels associated to problem goals and concepts. Five solving strategies can be employed by an agent to generate new knowledge. Group responses form a solution space, in which responses are grouped into categories based on their similarity and organized in abstraction levels. The solution space includes concrete features and samples, as well as the causal sequences that logically connect concepts with each other. The model was used to explain how member characteristics, e.g., the degree to which their knowledge is similar, relate to the solution novelty of the group. Model validation compared model simulations against results obtained through behavioral experiments with teams of human subjects, and suggests that TMs are a useful tool in improving the effectiveness of small teams.