Evaluation of Science Communication: Current Practices, Challenges, and Future Implications

Abstract

Scientifically substantiated evaluations are pivotal to ensuring the effectiveness and improvement of the growing number of science communication projects. Yet current evaluation practices are still lacking in various respects. Based on a systematic review of evaluation reports, an online survey of, as well as discussion rounds with science communication practitioners in the German-speaking countries, we discuss three main challenges of science communication evaluation: (1) There is a conflation of impact goals and measurable project objectives as well as a lack of precise definitions of objectives and target groups, which complicates the assessment of the projects' success. (2) Although many evaluations highlight the impact-oriented interest of those responsible, the methods chosen rarely allow scientifically valid evaluations of effects. The lack of comparative reference points and the partially unsuitable use of self-report measures are key issues in this regard. (3) The fact that few evaluation processes are made transparent and that formative evaluation designs are a rarity indicates a tendency to understand evaluations as the final ‘success story’ of a project rather than a learning process. This stands in the way of a constructive discussion of the actual impact of science communication. Our exploratory insights contribute to an understanding of the weaknesses of science communication evaluation and needs in the field. They also provide impulses for future improvements in the field for the stakeholders in practice, research, funding, and science management.

Imagining the Sun: using comparative judgement to assess the impact of cross-curricular solar physics workshops

“…The use of online comparative judgement could therefore be used to address issues in the quality of evaluation of science communication and STEM engagement activities identified by other researchers [Jensen, 2014;Ziegler et al, 2021] by providing a clear method of pre-and post-intervention evaluation.…”

“…In conclusion, the current paper indicates that the use of open-response questions and drawing with young children provides a helpful method of identifying changes in scientific knowledge before and after a cross-curricular workshop linking science and art. Following Ziegler et al [2021] we present the evaluation as a way of sharing learning with the science communication community. We found that the use of online comparative marking to rank children's responses allows a straight-forward method for data analysis which can be used by science communication and engagement project teams who may not have a background in qualitative methods.…”

“…However, the quality of evaluation of science communication and engagement activities can be low with a lack of strategic planning, vaguely targeted interventions and poor design. [Ziegler, Hedder & Fischer, 2021;Jensen, 2014;Morgan & Kirby, 2016]. This quality of evaluation may arise from lack of training for researchers who wish to take part in science communication [Davies, 2013].…”

Kommunikation: Wer ist die „breite Öffentlichkeit“?

“…Dafür brauchen wir im Wissenschaftssystem mehr Reflexion darüber, was Wissenschaftskommunikation bei wem erreichen soll. Und es braucht mehr kritische Betrachtungen der aktuellen Praxis durch Forschung und Evaluation, 11)…”

Communication and Transparency of Innovation: The Case of the Valencian Community (Spain)

“…This moderate impact could be due to the lack of benchmarks on effective communication. There is virtually no access to evaluations of scientific communication, and there are virtually no success stories or examples of ineffective communication that would allow organisations to learn from the experience [45].…”