Cyclic prefix orthogonal frequency division multiplexing (CP-OFDM) is the most widespread multicarrier modulation scheme. However, the use of a rectangular filter in CP-OFDM causes poor outof-band (OoB) radiation. Moreover, the CP causes spectral efficiency loss. To overcome these limitations, a filter bank multicarrier (FBMC) was proposed which uses prototype pulse shaping filters that can be adapted to fulfil the system requirements. The choice of the filter is crucial for FBMC/offset quadrature amplitude modulation (OQAM) due to its significant impact on the achieved performance. In this paper, new prototype pulse shaping filters in FBMC/OQAM systems are proposed, aiming to improve the system performance. A number of prototype pulse shaping filters, such as raised cosine pulse (RC), root raised cosine pulse (RRC), better than raised cosine pulse (BTRC), modified Bartlett-Hanning (MBH), improved sinc power pulse (ISP), phase modified sinc pulse (PMSP), parametric linear pulse (PLP), linear combination pulse (LCP), PHYDYAS and Hermite, are considered. The performance of each pulse shaping filter in FBMC/OQAM is evaluated and compared other in terms of the power spectral density (PSD), spectral efficiency, signal-to-interference ratio (SIR), time offset (TO), carrier frequency offset (CFO) and bit error rate (BER) performance over various channels. In addition, as multiple input multiple output (MIMO) is one of the main challenges associated with FBMC, we propose the new approach that use the frequency/time block spreading based on a Walsh-Hadamard (WH) code in MIMO FBMC/OQAM and investigate the combination of the our proposed prototype pulse shaping filters with MIMO systems. The proposed filters are shown to be suitable candidates for FBMC/OQAM systems.