Neutralizing antibody (NAb) is a family of antibodies with special functions, which afford a degree of protection against infection and/or reduce the risk of clinically severe infection. Receptor binding domain (RBD) in the spike protein of SARS-CoV-2, a portion of the S1 subunit, can stimulate the immune system to produce NAb after infection and vaccination. The detection of NAb against SARS-CoV-2 is a simple and direct approach for evaluating a vaccine’s effectiveness. In this study, a direct, rapid, and point-of-care bicolor lateral flow immunoassay (LFIA) was developed for NAb against SARS-CoV-2 detection without sample pretreatment, and which was based on the principle of NAb-mediated blockage of the interaction between RBD and angiotensin-converting enzyme 2. In the bicolor LFIA, red and blue latex microspheres (LMs) were used to locate the test and control lines, leading to avoidance of erroneous interpretations of one-colored line results. Under the optimal conditions, NAb against SARS-CoV-2 detection carried out using the bicolor LFIA could be completed within 9 min, and the visible limit of detection was about 48 ng/mL. Thirteen serum samples were analyzed, and the results showed that the NAb levels in three positive serum samples were equal to, or higher than, 736 ng/mL. The LM-based bicolor LFIA allows one-step, rapid, convenient, inexpensive, and user-friendly determination of NAb against SARS-CoV-2 in serum.