It is well known that rerankers built on pretrained transformer models such as BERT have dramatically improved retrieval effectiveness in many tasks. However, these gains have come at substantial costs in terms of efficiency, as noted by many researchers. In this work, we show that it is possible to retain the benefits of transformer-based rerankers in a multi-stage reranking pipeline by first using feature-based learning-to-rank techniques to reduce the number of candidate documents under consideration without adversely affecting their quality in terms of recall. Applied to the MS MARCO passage and document ranking tasks, we are able to achieve the same level of effectiveness, but with up to 18× increase in efficiency. Furthermore, our techniques are orthogonal to other methods focused on accelerating transformer inference, and thus can be combined for even greater efficiency gains. A higher-level message from our work is that, even though pretrained transformers dominate the modern IR landscape, there are still important roles for "traditional" LTR techniques, and that we should not forget history.1 Muppets being a whimsical way to refer to BERT and related transformer models.

Purpose: The residency interview is the most important factor for residency program directors when deciding on how to rank medical student applicants. With the residency match becoming increasingly competitive, it is more important than ever for students to perform well in this high-stakes interview. Video-stimulated recall (VSR) has been shown to be an effective tool for facilitating reflection on performance and behaviors. As such, we conducted mock interviews with and without video-stimulated recall to gauge its effect on student perceptions of preparedness and confidence for residency interviews. Methods: Students completed a pre-mock interview survey followed by a video recorded interview with faculty. All students received verbal feedback on their performance immediately after the interview. Students were randomized to receive their feedback from their faculty interviewer either while reviewing their video or without the video review. Post-mock interview and post-residency interview surveys were completed. Wilcoxon signed-rank was used to compare median aggregate scores between pre/post surveys. Wilcoxon rank-sum was used to compare pre/post aggregate scores between the video review vs. no-video review groups. Results: 33 of 70 students participated (47%). 14 students (42%) reviewed their video and 19 (58%) received feedback without video. Likert scores for pre-and post-mock interview and post-residency interview surveys revealed median aggregate scores of 10 (interquartile range, or IQR=8,11), 12 (IQR=12,13), and 13 (IQR=12,13) (p <0.001, p<0.001). The change in median aggregate score between pre/post-mock interview surveys in the video review group vs. no-video review group was 3 (IQR=3,5) and 1 (IQR=0,3) (p<0.01) and from pre-mock interview to post-residency interview in the video review vs. no-video review groups was 3 (IQR=3,5) and 2 (IQR=1,4) (p=0.04). Conclusions: The mock interview for residency application improved students' perceptions of preparedness and confidence. Reviewing the video of the interview while receiving verbal feedback increased students' confidence in their interview skills.

No abstract