The ball and beam (BnB) system serves as a benchmark in control engineering as it provides a foundational concept applicable to addressing stabilization challenges of various underactuated nonlinear systems. This includes tasks like maintaining the balance of goods carried by mobile robots and controlling the attitude of unmanned aerial vehicles. In this study, the focus is on enhancing control optimization strategies for BnB systems that take into account inherent nonlinearities arising from actuator constraints and state measurements. The work introduces a novel intelligent control approach, termed hybrid PSO-WOA, which combines Particle Swarm Optimization (PSO) and Whale Optimization Algorithm (WOA) to automate optimal parameter search for proportional-integral-derivative (PID) and state feedback (SF) controllers. The collaborative technique between PSO and WOA is formulated to strike a balance between exploration and exploitation phases, and to mitigate premature convergence risks due to the system's complexities. Additionally, three control schemes, namely cascade PID-PID, cascade PID-SF, and cascade PID-observer are introduced, each with tailored cost functions for optimization through the hybrid PSO-WOA algorithm, accommodating both measurable and unmeasurable state scenarios. Simulation results consistently demonstrate the superior performance of the hybrid approach compared to individual PSO and WOA methods, as well as conventional PID and linear quadratic regulator approaches. Notable, the hybrid approach exhibits a significant improvement in error metrics, reducing integral-time absolute error by 18.99%, integral squared error by 35.37%, and steady-state error by 92.86%. This substantial enhancement suggests promising directions for future research in automated control parameter tuning for underactuated nonlinear systems.