Abrin and ricin, both type II ribosome-inactivating proteins, are toxins of significant concern and are under international restriction by the Chemical Weapons Convention and the Biological and Toxin Weapons Convention. The development of a rapid and sensitive detection method for these toxins is of the utmost importance for the first emergency response. Emerging rapid detection techniques, such as surface-enhanced Raman spectroscopy (SERS) and lateral flow assay (LFA), have garnered attention due to their high sensitivity, good selectivity, ease of operation, low cost, and disposability. In this work, we generated stable and high-affinity nanotags, via an efficient freezing method, to serve as the capture module for SERS-LFA. We then constructed a sandwich-style lateral flow test strip using a pair of glycoproteins, asialofetuin and concanavalin A, as the core affinity recognition molecules, capable of trace measurement for both abrin and ricin. The limit of detection for abrin and ricin was 0.1 and 0.3 ng/mL, respectively. This method was applied to analyze eight spiked white powder samples, one juice sample, and three actual botanic samples, aligning well with cytotoxicity assay outcomes. It demonstrated good inter-batch and intra-batch reproducibility among the test strips, and the detection could be completed within 15 min, indicating the suitability of this SERS-LFA method for the on-site rapid detection of abrin and ricin toxins.