The negative effects of environmental estrogens on wildlife and human beings are gaining increasing attention. Research on the highly sensitive detection method for Vitellogenin (Vtg), one of the biomarkers of environmental estrogens (EEs), is expected to detect weak estrogens in complex environments. This study aimed to develop a label-free immunosensor with high specificity and sensitivity for testing Vtg. Carbon quantum dots (CQDs) with high fluorescence and excellent stability were synthesized, and antilipovitellin monoclonal antibody (Anti-Lv-mAb) was prepared. Based on the fluorescence resonance energy transfer (FRET) between CQDs-conjugated Anti-Lv-mAb and reduced graphene oxide (RGO), an ultrasensitive fluorescent “ON-OFF” label-free immunosensor for detection of Vtg of marine medaka was established. By modification of RGO with poly dimethyl diallyl ammonium chloride (PDDA), the Zeta potential of RGO was changed and the FRET efficiency was improved. The immunosensor displayed a wide linear response to Vtg of marine medaka from 0.1 to 3000 ng/mL, a low limit of detection (LOD) of 0.04 ng/mL, and excellent sensitivity (28,833.63 CPS/(ng/mL)), selectivity, and reproducibility. The results demonstrated that the fluorescent “ON-OFF” immunosensor is an easy-to-use, relatively fast, ultrasensitive, and accurate detection method for weak estrogenic activity.