Systematics in the ALMA Proposal Review Rankings

Abstract

The results from the ALMA proposal peer review process in Cycles 0-6 are analyzed to identify any systematics in the scientific rankings that may signify bias. Proposal rankings are analyzed with respect to the experience level of a Principal Investigator (PI) in submitting ALMA proposals, regional affiliation (Chile, East Asia, Europe, North America, or Other), and gender. The analysis was conducted for both the Stage 1 rankings, which are based on the preliminary scores from the reviewers, and the Stage 2 rankings, which are based on the final scores from the reviewers after participating in a face-to-face panel discussion. Analysis of the Stage 1 results shows that PIs who submit an ALMA proposal in multiple cycles have systematically better proposal ranks than PIs who have submitted proposals for the first time. In terms of regional affiliation, PIs from Europe and North America have better Stage 1 rankings than PIs from Chile and East Asia. Consistent with Lonsdale et al. (2016), proposals led by men have better Stage 1 rankings than women when averaged over all cycles. This trend was most noticeably present in Cycle 3, but no discernible differences in the Stage 1 rankings are present in recent cycles. Nonetheless, in each cycle to date, women have had a lower proposal acceptance rate than men even after differences in demographics are considered. Comparison of the Stage 1 and Stage 2 rankings reveal no significant changes in the distribution of proposal ranks by experience level, regional affiliation, or gender as a result of the panel discussions, although the proposal ranks for East Asian PIs show a marginally significant improvement from Stage 1 to Stage 2 when averaged over all cycles. Thus any systematics in the proposal rankings are introduced primarily in the Stage 1 process and not from the face-to-face discussions. These results are discussed in the context of potential language and cultural biases, but any conclusions on the origin of the observed systematics remain speculative. Lonsdale et al. (2016) analyzed the results from the proposal review process for four facilities operated in full or in part by the National Radio Astronomical Observatory (NRAO): the Jansky Very Large Array (JVLA), the Very Long Baseline Array (VLBA), the Green Bank Telescope (GBT) and ALMA. Analogous to the results for HST and ESO, they found that the proposal rankings favored men over women in ALMA Cycles 2-4, with the largest and most significant difference found in Cycle 3. The other NRAO telescopes showed similar trends, although the significance was lower than found for ALMA, and in some semesters, women had higher overall rankings than men. Hunt et al. (2019) extended the analysis by Lonsdale et al. (2016) to include more recent proposal rounds at the JVLA, VLBA, and the GBT. They found that when averaged over all proposal semesters between 2012A and 2019A for the JVLA, VLBA and GBT combined, men had a statistically significant advantage over women in the proposal scores.

Gender Systematics in the NRAO Proposal Review System

“…This was first discovered in HST observing proposals where male PIs had better success rates than female PIs (Reid 2014). Similar trends have been found at ESO (Patat 2016), ALMA (Lonsdale et al 2016;Carpenter 2020), NRAO (Lonsdale et al 2016), and the Canada-France-Hawaii Telescope (CFHT) and Gemini Observatory (Spekkens et al 2018).…”

“…Other effects such as prestige bias complicate the interpretation of the data. For example, some studies have noted a dependence on the gender-based success rate and PI seniority (Spekkens et al 2018); the seniority of the review panels (Patat 2016); and the potential privilege of a proposer being a member of the TAC (Greaves 2018)-although Carpenter (2020) did not find any correlation. Numerous studies have noted that both males and females evaluate work by males higher than females for similar results.…”

“…The data to study the seniority of PIs was not available. Carpenter (2020) expanded the gender systematics study of ALMA to cycles 0-6. The analysis considered both review stage 1, which includes preliminary scores, and review stage 2, which are final scores after a face-to-face discussion within each panel.…”

Update on the Systematics in the ALMA Proposal Review Process after Cycle 8

“…In this paper, we update the analysis presented in Carpenter (2020) and investigate any changes in the systematics in the past two ALMA cycles (Cycle 7 and Cycle 8). Section 2 provides an overview of the ALMA proposal review process.…”

Inequalities faced by women in access to permanent positions in astronomy in France

“…Vera Rubin, that access to Mount Wilson observatory for women was a struggle [4]. Recent empirical studies have shown that gender biases in the attribution of telescope time persist at major facilities such as the Atacama Millimeter Array [5], the Hubble Space Telescope [6], the European Southern Observatory [7], and the NRAO [8]. Although women are now better represented in large scientific collaborations in astronomy, only few of them serve as leaders in these teams [9], and when it gets to receiving a share of a prize, they can be forgotten [10].…”