Background Rheumatic heart disease affects 33 million people in low and middle income countries and is the leading cause of cardiovascular death among children and young adults. Penicillin prophylaxis has been shown to improve valvular function among patients with clinically silent or mild disease. Efforts to expand echocardiographic screening are focusing on simplified protocols, non-physician ultrasonographers, and portable ultrasound devices, including handheld ultrasound. Recent advances support the use of single-view screening protocols. With the increasing availability and low cost of handheld devices, prospective studies are needed to evaluate their performance in these settings. Methods We conducted a cross-sectional study among 19 at-risk school-children participating in a rheumatic heart disease screening program in Ethiopia comparing a handheld ultrasound device (Phillips Lumify) to a fully-equipped portable ultrasound machine (Sonosite M-Turbo). Results Agreement between devices was similar for expert and non-expert review (84%). However, when reviewed by a non-expert the Lumify identified fewer screen-positive cases (p-value 0.083). We also compared non-expert to expert interpretation by device and found a significant difference in interpretation for the Lumify (p-value 0.025). There was a trend towards shorter jet length by color Doppler in the handheld ultrasound device for both expert and non-expert review. Conclusions Our study highlights the importance of using caution when applying proposed single-view criteria as the sensitivity and specificity may be affected by the device.