Aflatoxins (AFs) are highly toxic compounds that can cause both acute and chronic toxicity in humans. Aflatoxin B1 (AFB1) is considered the most toxic of AFs. Therefore, the rapid and on-site detection of AFB1 is critical for food safety management. Here, we report the on-site detection of AFB1 in grains by a portable surface plasmon resonance (SPR) sensor. For the detection of AFB1, the surface of an SPR Au chip was sequentially modified by cysteine-protein G, AFB1 antibody, and bovine serum albumin (BSA). Then, the sample solution and AFB1-BSA conjugate were flowed onto the Au chip in serial order. In the absence of AFB1, the SPR response greatly increased due to the binding of AFB1-BSA on the Au chip. In the presence of AFB1, the SPR response showed little change because the small AFB1 molecule binds on the Au chip instead of the large AFB1-BSA molecule. By using this portable SPR-based competitive immunoassay, the sensor showed low limits of detection (2.51 ppb) and quantification (16.32 ppb). Furthermore, we successfully detected AFB1 in rice, peanut, and almond samples, which suggests that the proposed sensing method can potentially be applied to the on-site monitoring of mycotoxins in food.