Maximizing spectral efficiency and capacity of the conventional radio frequency (RF) systems are critical challenges occurring especially in RF-based indoor wireless communications where high data rate demands arise. Hence, the urge for finding an alternative wireless carrier is the key for exploiting the visible light communication (VLC) bands. However, the main issue incorporated with the usage of VLC band is the high dependency of connectivity on line of sight (LOS). Therefore, hybrid RF/VLC system has been top-hot topic in wireless communication where RF is capable of offering wide-area coverage and VLC has the ability to provide high transmission data rate. This paper presents vertical handover-based hybrid RF/VLC intra-software defined wireless body area network architecture to guarantee the different QoS requirements. Our proposed scheme maintains network reliability by prioritizing the processing of the emergency data. The proposed hard switching algorithm is formulated as multi-objective optimization problem aiming at minimizing the outage and blocking probabilities, worst-case latency, packet error rate, and maximizing the throughput and system fairness. To solve such multi-optimization problem, we have used Pareto simulated annealing to efficiently select either VLC link or RF link satisfying the system constraints. Simulation results show that the proposed hybrid RF/VLC scheme satisfies QoS for WBANs. In comparison with the standalone VLC system, hybrid RF/VLC scheme outperforms the latter for smaller values of signal to interference plus noise ratio. Better network performances were recorded in comparison with other state-of-the-art schemes.