A new method is proposed in this paper which designs a reduced-order observer of a non-observable-form-based dynamical system such that: (i) the eigenvalues are specified to satisfy desired convergence performance, (ii) a full-rank condition is satisfied, and (iii) a quadratic performance measurement of the deviation of the estimates from the actual states is minimized so as to reduce the large error occurring during the transient period of observation. The proposed approach combines the merits of both the orthogonal functions approach and evolutionary optimization. By solving a Sylvester equation, the proposed optimal design method can not only be used to design the reduced-order observer of a non-observable-form-based dynamical system, but also can avoid the shortcomings of approaches already existing in relevant literatures. Two examples are given to demonstrate the effectiveness and efficiency of the proposed new optimization method on the performance of state estimations. From the demonstrative examples, it can be seen that the estimated state errors asymptotically converge quickly to zero. In addition, the performance measurement values based on the proposed optimal design approach are apparently much lower than those based on the existing nonoptimal design method.