Different barrow and gilt responses were observed during HAT and NOT. During HAT, barrows spent 2% more time within zone 1 (P = 0.03), crossed fewer zone lines (P < 0.0001), changed head orientation less frequently (P = 0.002), and froze less frequently compared to gilts (P = 0.02). However, during NOT, barrows froze more frequently (P = 0.0007) and spent 2% longer freezing (P = 0.05). When the behavior and RFI relationship was examined using odds ratios, decreasing RFI by 1 kg/d decreased the odds of freezing by 4 times but increased the odds of attempting to escape by 5.26 times during NOT (P ≤ 0.04). These results suggest that divergent selection for RFI resulted in subtle behavioral reactivity differences and did not impact swine welfare in its response to fear-eliciting stimuli.