Update on the Systematics in the ALMA Proposal Review Process after Cycle 8

Abstract

We present an updated analysis of systematics in the Atacama Large Millimeter/submillimeter Array (ALMA) proposal ranks from Carpenter (2020) to include the last two ALMA cycles, when significant changes were introduced in the proposal review process. In Cycle 7, the investigator list on the proposal cover sheet was randomized such that the reviewers were aware of the overall proposal team but did not know the identity of the principal investigator (PI). In Cycle 8, ALMA adopted distributed peer review for most proposals and implemented dual-anonymous review for all proposals, in which the identity of the proposal team was not revealed to the reviewers. The most significant change in the systematics in Cycles 7 and 8 compared to previous cycles is related to the experience of PIs in submitting ALMA proposals. PIs that submit a proposal every cycle tend to have ranks that are consistent with average in Cycles 7 and 8 whereas previously they had the best overall ranks. Also, PIs who submitted a proposal for the second time show improved ranks over previous cycles. These results suggest some biases related to the relative prominence of the PI have been present in the ALMA review process. Systematics related to regional affiliation remain largely unchanged in that PIs from Chile, East Asia, and non-ALMA regions tend to have poorer overall ranks than PIs from Europe and North America. The systematics of how one region ranks proposals from another region are also investigated. No significant differences in the overall ranks based on gender of the PI are observed.

Analysis of the ALMA Cycle 8 Distributed Peer Review Process

“…In all cases, the trends explored in this section are suggestive and reported for completeness, but given that the number of assignments which fell into Rules 3-20 is less than 10% of all assignments, the impact on the final rankordered list is not significant. However, these trends -as well as the systematic trends examined in Carpenter et al (2022) -will be monitored by the JAO in upcoming Cycles.…”

“…In this section, we examine the systematics in the rankings depending on the science category and popularity of the keywords, investigate the dispersion in the ranks, address the impact of outlier rejection, and compare the Cycle 8 results to those from the Cycle 7 panel reviews. Systematics in the rankings related to gender, regional affiliation of the PI and reviewer, and experience level of the PI are examined in Carpenter et al (2022).…”