Recent partisan claims about the illegitimacy of the 2020 election highlight a need to better understand the determinants underlying Americans' trust in the electoral process. In this study, we focus on African Americans and conservatives, two groups that stand out both historically and contemporaneously for high levels of distrust in elections. Utilizing nationally representative survey data, we analyze the degree to which election distrust is associated with respondents’ attitudes towards policies addressing voter access (photo ID requirements, vote by-mail, and felon voting), perceptions of disenfranchisement, and their personal experience while voting. We find evidence that distrust is not tied to one’s personal voting experience, but rather to one’s policy attitudes towards electoral access. Importantly, for conservative and Black voters, the policy remedies that would lead to increased trust for one group would only further exacerbate the concerns of the other, suggesting that distrust towards American elections will be difficult to attenuate.