Although access to scientific information has improved for the general public since the introduction of plain language summaries (PLSs) and the open-access publishing movement, language barriers still impede the widespread dissemination of information. Most scientific articles are published in English language only, despite English speakers comprising just 17% of the world’s population. Here we present a pilot analysis that aimed to compare the translation quality of PLSs and abstracts translated by a selected browser-based translation software. We translated abstracts and PLSs from 5 medical journal publications into French, German, Mandarin, and Slovenian using Google Translate. Four bilingual reviewers with a scientific background assessed the translation quality using pre-defined survey questions that covered the appropriateness of word/phrase selection, grammar, and clarity. We assessed the number of errors of each type and used a 5-point Likert scale to measure the impact of these errors on the meaning of the text. Translations of both PLSs and abstracts were considered accurate and readable, although PLSs scored higher across most measures. For overall accuracy, translated PLSs scored higher on the Likert scale than translated abstracts (mean, 4.60 vs 4.30, respectively), with 60% of PLS translations considered to be “very accurate” compared with 45% of abstract translations. PLSs were also considered less likely to be misinterpreted (mean, 4.55 vs 4.25, respectively), with 60% of PLS translations versus 45% of abstract translations reported as “definitely not” likely to be misinterpreted. Based on our findings, Google Translate potentially offers a quick and easy approach to translating scientific/medical information summaries for non-English speakers. However, before these articles can be translated, they must be discoverable by non-English speakers. Engagement and collaboration with medical publishers are needed to improve access for non-English speakers, including provision and PubMed indexing of PLSs that can be translated easily.