This study presents a new property of the filter bank-based multicarrier (FBMC) system. Also, an efficient iterative algorithm for designing the system with a large number of subcarriers and a prototype filter with a very long length are proposed. For the system, the compact from conditions are derived for both the intersymbol interference free and the interchannel interference (ICI) free. Based on these new conditions, the design of the prototype filter is formulated as an unconstrained optimisation problem where the objective function is the weighted sum of the total distortion of the system and the stopband energy. By deriving the gradient vector of the objective function, an efficient iterative algorithm is proposed for finding the solution of the optimisation problem. In addition, an efficient matrix inversion approach is presented to greatly reduce the computational complexity of the iterative algorithm. As a result, it is feasible to design the FBMC system with thousands of subcarriers. The convergence of the iterative algorithm is proved. Computer numerical simulation results with the comparisons to the existing methods are presented. It is shown that the proposed design algorithm is more effective and efficient than the existing methods.