Recently, the finite state machine (FSM)-based digital controllers are susceptible to faultinjection and side-channel attacks, which makes FSM security a more prominent factor. FSM optimality is another crucial element when designed. A state encoding approach is applied for FSM security and optimization. This article proposes the security-dominated FSM state assignment technique (SD-SHO), which obtains a satisfactory level of FSM optimization as well. It is a deterministic algorithm and consists of two key techniques, such as state assignment using an improved quadratic sum code, and state assignment using a gradient-based interior point method. A fuzzy bi-level programming logic is introduced in the proposed approach for regulating the constituting algorithms optimally. Experiments are conducted to evaluate the FSM security and optimality using the MCNC FSM benchmarks. Results indicate a substantial reduction in the error masking probability using SD-SHO. It also demonstrates that SD-SHO achieves a satisfactory level of area and power reduction compared with other existing works.This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.