Voice quality differences [1] can convey different attitudes and emotions [2], with speakers of different languages showing different sensitivities to voice qualities, e.g., [3,4,5]. It remains to be explored, however, precisely which acoustic properties are perceptually associated with what emotional meanings, and whether such perceptual mappings hold universally or differ across languages. This paper offers a first step addressing these issues. Building upon the previous findings that speakers of different languages demonstrate different sensitivities to voice quality differences, the study examines particularly how the perceptions of arousal and valence are affected by different voice qualities. The current experiment reveals that speakers of the three language groups share similar ratings of arousal in association with breathy voices. Yet the valence ratings vary among the groups: Japanese and Mandarin listeners rate voices with high F0 and small OQ with positivity, whereas Brazilian Portuguese rate voices with low F0 and larger OQ with positivity. The findings of this study have applications for second language teaching, and carry over to the worlds of business, politics, and advertisement; in general, this type of research may have a potential to be useful for improving communication in crosscultural inter-personal relationships.