In this article, we study a kind of linear quadratic optimal control problem driven by forward-backward stochastic differential equations (FBSDEs in short) with deterministic coefficients. The cost functional is defined by the solution of FBSDEs. By means of the Girsanov transformation, the original issue is turned equivalently into the classical LQ problem. By functional analysis approach, some necessary and sufficient conditions for the existence of optimal controls have been obtained. Moreover, we investigate the relationship between two groups of first-order and second-order adjoint equations. A new stochastic Riccati equation is derived, which leads to the state feedback form of optimal control. By introducing a new Hamiltonian function with an exponential factor, we establish the stochastic maximum principle to deal with the stochastic linear quadratic problem for forward-backward stochastic system with nonconvex control domain using first-order adjoint equation. An illustrative example is given as well.
K E Y W O R D Sforward-backward stochastic differential equations, Hilbert method, maximum principle, stochastic linear quadratic optimal control the literature called a linear quadratic optimal control Problem (LQ problem, in short).