This study investigates the extent to which scholarly tweets of scientific papers are engaged with by Twitter users through four types of user engagement behaviors, i.e., liking, retweeting, quoting, and replying. Based on a sample consisting of 7 million scholarly tweets of Web of Science papers, our results show that likes is the most prevalent engagement metric, covering 44% of scholarly tweets, followed by retweets (36%), whereas quotes and replies are only present for 9% and 7% of all scholarly tweets, respectively. From a disciplinary point of view, scholarly tweets in the field of Social Sciences and Humanities are more likely to trigger user engagement over other subject fields. The presence of user engagement is more associated with other Twitter-based factors (e.g., number of mentioned users in tweets and number of followers of users) than with science-based factors (e.g., citations and Mendeley readers of tweeted papers). Building on these findings, this study sheds light on the possibility to apply user engagement metrics in measuring deeper levels of Twitter reception of scholarly information.