A vector polarized beam undergoing a cyclic adiabatic transformation can obtain a Pancharatnam-Berry (PB) phase. This new geometric phase is proportional to light's total angular momentum (TAM). In this work, experimental evidence is provided for the PB phase evolution on a hybrid-order Poincaré sphere. A variety of hybrid-order Poincaré sphere beams are directly generated by vortex half-wave plates using a non-interference method, while two TAM converters are used to ensure a closed cycle on the hybrid-order Poincaré sphere. By rotating one of the converters, a significant change in PB phase can be observed. The PB phase is related to the topological charge and polarization state of the hybrid-order Poincaré sphere bipolar beams, and has implications in quantum information science as well as other physical systems such as electron vortex beams.