A consumer study was conducted to determine if consumers scored beef palatability traits differently on paper vs. digital ballots. Beef subprimals were collected to represent four treatments with inherent variation in eating quality: USDA Select eye round aged 7 d postmortem, USDA Select strip loin aged 7 d postmortem, USDA Choice tenderloin aged 21 d postmortem, and USDA Prime strip loin aged 21 d postmortem. Accessory muscles, external fat, and connective tissue were removed from subprimals. Muscles were fabricated into 2.5 cm steaks and further divided into 2 equal halves for consumer testing. Consumers (n = 360) were served eight samples divided into two blocks representing the two ballot types. Select longissimus lumborum samples were always served in the first and fifth position to start each ballot block, with the remaining three treatments served in a randomized order between the latter three positions within each ballot block. Consumers rated each steak sample for tenderness, juiciness, flavor liking, and overall liking on either a paper or digital ballot, and then rated the paired steak halves on the opposite ballot during the second block of sample testing. Ballot type influenced (P < 0.02) all traits, as consumers scored tenderness, juiciness, flavor liking, and overall liking greater (P < 0.05) on paper ballots compared to digital ballots, regardless of treatment. The smallest margin between paper and digital ballots was observed for tenderness, where scores only differed by 1.8 units. However, juiciness, flavor liking, and overall liking all differed by 3.4 units in favor of paper ballots. Consumers scored palatability traits greater on paper ballots compared to digital ballots, and these findings were consistent across a range of eating quality outcomes. These results should be considered when comparing past consumer testing results using paper balloting systems to those where digital ballots were used.