The aim of this study was to investigate the impact of signboard‐building color combinations on color harmony and legibility. Two hundred and three participants rated 54 signboard‐building color combinations against two scales of color harmony and legibility. In this article, the terms “brick,” “stone,” and “glass” refer to three types of building exteriors used in the experiment (ie, brick masonry, greystone, and curtain walls, respectively). Major findings are as follows: (a) there was a positive linear correlation between color harmony and legibility in all three types of building exteriors, (b) the type of building exterior affected the color harmony and legibility of signboard colors, (c) no hue‐related patterns were observed, (d) the effects of chroma differences on color harmony were weak and the effects of chroma differences on legibility were moderate, (e) the effects of lightness differences on color harmony and legibility were strong in brick, but the effects of lightness differences were weak in stone and glass, (f) white color combinations (ie, color pairs including white signboards) turned out to be the most harmonious and legible, and (g) color combinations of light signboards and dark buildings (negative polarity) were rated most harmonious and legible, with the exception of vivid red (positive polarity). The findings of this study provide insight into the characteristics of harmonious and legible colors in the context of signboard design.