As a highly used form of architecture, public facilities are closely related to people’s daily lives. The aesthetic level of their design greatly affects the quality of people’s lives and interactions, as well as the appearance of the whole area. However, research on their design elements has not received enough attention, and few quantitative studies on the design of public facilities simultaneously focus on multiple influencing factors such as color and material. Therefore, this study uses eye-tracking technology and audience aesthetic preference evaluation to explore the appropriate combination of color and materials in sign design. This study found that, in the design of public facilities and in terms of aesthetic preference, firstly, color has more influence on subjective preference and objective gaze behavior than material. Secondly, men prefer technological and changeable colors and materials, and women prefer soft and uniform materials. Finally, visitors spend more time gazing at their preferred signs, which means that visitors’ top-down gaze behavior and bottom-up psychological perception are highly unified. Regarding material perception, visitors’ subjective preferences are influenced by the physical or cultural properties of the material itself. This study’s results will provide suggestions for future urban and landscape architecture design in terms of color and material aesthetics, and the research methodology can be applied to more scenarios in environmental spaces.