Filtered-orthogonal frequency division multiplexing (F-OFDM) is a candidate for 5G new waveform, which has flexible numerologies to support various scenarios and can combat interference of multi-users effectively with filters. However, although longer filter length and higher roll-off factor can suppress out of band radiation, inter-symbol interference (ISI) and inter-carrier interference (ICI) will increase at the same time. Meanwhile, the wider guard band reduce inter-user interference (IUI) at the cost of higher spectrum occupancy. To meet target bit error rate (BER) requirements of all users and maximize the spectrum efficiency, a filter parameter and guard bandwidth optimization model was proposed to design related parameters dynamically for uplink asynchronous F-OFDM system, where users are allocated with different numerologies. We present a detailed theoretical derivation on ISI, ICI, and IUI for BER calculation, and the theoretical derivation of IUI in multi-user scenario with different subcarrier spacing between users is first presented. The formulated problem is proved to be a non-convex NP hard problem and Imperialist Competitive Algorithm (ICA) is applied for a desirable solution. The simulation results show that the theoretical derivations are valid and the proposed filter parameter and guard bandwidth optimization model can achieve higher spectrum efficiency than the fixed filter parameters in F-OFDM while outperforms traditional OFDM.INDEX TERMS Filtered-orthogonal frequency division multiplexing (F-OFDM), interference analysis, spectrum efficiency, multi-numerologies. MEIJIE YANG received the B.S. degree from the University of Science and Technology Beijing, China, in 2016, where she is currently pursuing the Ph.D. degree. Her research interests mostly include physical layer, advanced waveforms techniques, and signal processing for 5G systems. YUEYUN CHEN received the bachelor's degree from the South China University of Technology and the master's and Ph.D. degrees from Beijing Jiaotong University. She is currently a Professor with the School of Computer and Communication Engineering, University of Science and Technology Beijing, China. Her current research interests include wireless and mobile communications, including MIMO, communication signal processing, radio resource management, cognitive radio, cooperative communications, and optimization theory on communications. LIPING DU received the B.E. degree in industrial automation and the M.Sc. degree in physical sci-