Color plays an essential role in the design of user interfaces and significantly impacts the user experience. To create aesthetically pleasing and user-friendly interfaces, the colors of the user interface should be consistent with the images. The latter can be challenging to achieve, as images often have different colors and are often changed by editors or authors who do not have sufficient design knowledge. To solve this problem, we have developed a model that automatically determines the color theme of the user interface based on a given image. The model first extracts the most prominent colors from the image and then considers both aesthetic (color harmony and compatibility with the image) and usability aspects (color contrast, color diversity, and color strength). All color calculations are performed in the perceptually uniform color space CAM02-UCS. In addition, the model can be adapted to the user’s needs and requirements. To test the model, we implemented it in a web-based application in which the colors were automatically selected based on the featured image. The resulting color themes were then evaluated by the users, who were mainly professional designers. According to the results, the model generates color themes that are consistent with the image, aesthetic, and user-friendly. An important observation was also that color harmony can be achieved simply by using the most prominent colors of the image (regardless of their hue), suggesting that color harmony is strongly influenced by the context of use. The presented model holds significant practical importance as it can be utilized in various applications and tools. For instance, it can automatically choose a color theme for a user interface based on a particular image, such as a company logo or a product image. Moreover, it can dynamically adjust the colors of elements in real time based on the image that is visible simultaneously with the elements.