Massive multiple-input multiple-output (MIMO) can considerably enhance the "spectral efficiency and energy efficiency" since it is a major technique for future wireless networks. Thus, the performance needs a huge count of base station antennas to serve a smaller number of terminals in conventional MIMO methodology. Large-scale radio frequency (RF) chains represent the large-scale antennas. There is a need of implementing an effective massive MIMO system for maximizing the efficient performance of the system with high "spectral efficiency and energy efficiency" owing to the high cost of RF chains, and the higher power consumption. In this paper, a massive MIMO communication system is implemented to satisfy the requirements regarding "energy efficiency and spectral efficiency." Here, the number of base station antennas, the transmit power, and beam forming vectors are optimized to maximize "energy efficiency and spectral efficiency" when the channel capacity is known to be higher than some threshold values. The novelty of this work is a new hybrid optimization adaptive shark smell-coyote optimization (ASS-CO) algorithm is developed for improving energy efficiency. The optimization is done with the help of the hybrid optimization ASS-CO Algorithm. The proposed ASS-CO algorithm-based massive MIMO communication system is evaluated by experimental analysis. From the result analysis, the maximum resource efficiency is observed by SS-WOA, which is 6.6%, 50%, 6.6%, 6.6%, and 6.6% maximized than rider optimization algorithm (ROA), spotted hyena optimization (SHO), lion algorithm (LA), Shark Smell Optimization (SSO), and Coyote Optimization Algorithm (COA) by taking the count of base stations as 4. The superior performance enhancement regarding "spectral efficiency and energy efficiency" is accomplished over the traditional systems.