Public speaking is a communication ability that is expressed in social contexts. Public speaking anxiety consists of the fear of giving a speech or a presentation and the perception of being badly judged by others. Such feelings can impair the performance and physiological activation of the presenter. In this study, eighty participants, most naive in Virtual Reality experience, underwent one of four virtual reality public speaking scenarios. Four different conditions were tested in a between-group design, where the audience could express positive or negative non-verbal behavior (in terms of body gesture and facial expression), together with positive or adverse questions raised during a question-and-answer session (Q&A). The primary outcomes concerned the virtual audience behavior’s effect on perceived anxiety and physiological arousal. In general, perceived anxiety seemed to be unaffected neither by the verbal nor non-verbal behavior of the audience. Nevertheless, experimental manipulation showed a higher susceptibility to public speaking anxiety in those participants who scored higher on the Social Interaction Anxiety Scale (SIAS) than those with lower Social Interaction Anxiety Scale scores. Specifically, in the case where the verbal attitude was negative, high SIAS trait participants reported a higher level of anxiety. Participants’ physiological arousal was also affected by the proposed scenarios. Participants dealing with an approving audience and encouraging Q&A reported increased skin conductance response. The lack of correlation between reported anxiety and skin conductance response might suggest a physiological engagement in an interactive exchange with the virtual audience rather than a form of discomfort during the task.