Gyrotrons are the only RF sources providing significantly high output powers at continuous-wave operation in the sub-terahertz frequency range. A prerequisite for a proper study of gyrotron behavior is the accurate time-dependent self-consistent simulation of the interaction between the electron beam and the RF wave inside the interaction region (cavity). That requires an accurate description of the RF fields at the boundaries of the interaction region. In this work, an improved broadband boundary condition is proposed. Based on polynomial series expansion of the load impedance and of the wavenumber, this boundary condition not only features a significantly improved broadband matching, but also allows a customizable frequency-dependent reflection coefficient at the boundaries. The boundary condition has a general formulation, i.e., it does not rely on a special numerical method. For the case of perfect matching, the new formulation is validated through comparing the numerically calculated reflection coefficients with those obtained by the state-of-the-art matched gyrotron broadband boundary condition, whereas the validation in the case of a given frequency-dependent reflection is done by comparing the numerically calculated reflection with the prescribed one. Although this concept is initially intended for the simulation of gyrotron cavities, it can be easily extended to any other open-cavity resonators, for which proper definition of the boundary conditions is required.