Background: The early detection of COVID-19 patients is fundamental for containing the pandemic. A reverse-transcriptase quantitative polymerase chain reaction (RT-PCR), which detects SARS-CoV-2 RNA, is the gold standard diagnostic test, although it can contribute to false-negative results. Consequently, supplementary diagnostic tests are urgently needed. Methods: To assess the value of anti-SARS-CoV-2 antibody-based tests for confirming COVID-19, a retrospective study was conducted on 3120 inbound overseas travelers who underwent a 14-day government quarantine in Xiamen from August 2020 to October 2020. The diagnostic accuracy of the total antibody that detected the anti-SARS-CoV-2 antibody and the RT-PCR that detected SARS-CoV-2 RNA was determined in comparison to the clinical diagnosis. Results: The COVID-19 positive rate was 3.14% (98/3120). The sensitivity and specificity of the RT-PCR test on the first day of quarantine were 14.29% and 100%, respectively, and the sensitivity and specificity of the total antibody were 93.88% and 99.40%, respectively. The kappa value between an RT-PCR on the first day of quarantine and a clinical diagnosis was 0.24 (95% CI, 0.14–0.35), indicating poor consistency. The kappa value between total antibodies and a clinical diagnosis was 0.88 (95% CI, 0.83–0.93), indicating perfect consistency. There were no differences in the positive rates of an RT-PCR in symptomatic COVID-19 (7.41% (2/27)) and asymptomatic COVID-19 (16.90 (12/71) (p = 0.338). Similarly, the positive rate of the total antibody tests showed no difference in symptomatic COVID-19 (96.30% (26/27)) and asymptomatic COVID-19 (92.96% (66/71)) (p = 0.676). Conclusion: SARS-CoV-2 antibodies are developed by the body in response to an infection or after vaccination; this can easily lead to a missed diagnosis. In the context of low sensitivity for an RT-PCR, SARS-CoV-2 antibody detection is an effective adjunct to RT-PCR detection, which can improve the diagnostic accuracy of COVID-19 and provide an effective complement to the false-negative results of an RT-PCR.