Report on the first workshop on negative and null results in eScience

Massivizing Computer Systems: a Vision to Understand, Design, and Engineer Computer Ecosystems through and beyond Modern Distributed Systems

“…We envision fostering a domain of MCS where everything we develop is tested and benchmarked, reproducibly. Although providing a full set of principles leading to this goal goes beyond the scope of this article 40 , we see a set of desirable steps toward this end: (i) Reproducibility as essential service to the community: we must mature as a science and value reproducibility studies [76], including by publishing reproducibility studies as other domains do [77]; (ii) Open-access, open-source: both software [60] and data artifacts are shared with all stakeholders, receiving for this just reward and recognition [78], including appropriate levels of funding; (iii) Negative results are useful: following an increasingly visible community in Software Engineering [79], we postulate that past failures, especially observed through experiments that falsify predicted results, must be recorded and shared, leading to future success; (iv) Neutral results are useful: in the current approach of the science of computer systems, it seems that results are rarely worthy of publication, unless the results are strongly positive (or, rarely, strongly negative). We envision that neutral, even if previously unknown and expanding the body of knowledge on meaningful problems 41 , will receive as much opportunity for publication as the other kinds of results; (v) Laws and theories of ecosystem operation are valuable: contrasting to what we perceive as a bias toward "working systems", we see an increasing need for conducting empirical and other forms of research leading to laws of operation and possibly theories derived from it.…”