In this paper, a mathematical model was used to evaluate a dynamical hybrid system for optimizing and controlling the efficacy of plant-based protein in aquafeeds. Fishmeal (FM), raw rapeseed meal (RM), and a fermented meal with yeast (RM-Yeast) and fungi (Aspergillus oryzae RM-Koji) were used as test ingredients for the determination of apparent digestibility coefficients (ADCs) of dry matter, crude protein, crude lipid, energy, and essential amino acids (EAA) for olive flounder (Paralichthys olivaceus, 7 ± 0.02 g) using diets containing 0.5% Cr2O3 as an inert indicator. Among all ingredients tested, FM had the maximum ADC of dry matter (
P
<
0.05
), protein (
P
<
0.05
), lipid (
P
>
0.05
), and energy (
P
>
0.05
). Fermented meals (RM-Yeast and RM-Koji) showed higher ADC (
P
<
0.05
) of crude protein compared with RM, while there was no significance in ADCs of crude lipid and energy among different forms of rapeseed meal. Besides, ADC of crude lipid for RM-Yeast and RM-Koji, on the one hand, and ADC of gross energy for RM-Yeast, on the other hand, were not varied from that for FM (
P
>
0.05
). Amino acid digestibility reflects protein digestibility in most cases. Interestingly, protease, lipase, and amylase activities were better expressed in RM-Koji, RM-Yeast, and FM over RM, respectively. The current results deliver important information on nutrients and energy bioavailability in raw and fermented RM, which can be implemented to accurately formulate applied feeds for olive flounder. Compared with other applicable systems, the complexity of the approach implemented has been considerably reduced.