This paper presents a new control strategy that combines classical control and an optimization scheme to regulate the output voltage of the bidirectional converter under the presence of matched and mismatched disturbances. In detail, a control-oriented modeling method is presented first to capture the system dynamics in a common canonical form, allowing different disturbances to be considered. To estimate and compensate for unknown disturbances, an extended state observer (ESO)-based continuous sliding mode control is then proposed, which can guarantee high tracking precision, fast disturbance rejection, and chattering reduction. Next, an extremum seeking (ES)-based adaptive scheme is introduced to ensure system robustness as well as optimal control effort under different working scenarios. Finally, comparative simulations with classical proportional-integral-derivative (PID) control and constant switching gains are conducted to verify the effectiveness of the proposed adaptive control methodology through three case studies of load resistance variations, buck/boost mode switching, and input voltage variation.