This paper compares the nonlinear dynamics of two key types of motion observed in a rotating liquid-filled cavity subject to external forcing: an inertial wave attractor and resonant inertial oscillations (inertial modes). Experiments are performed with a cavity having a specific shape of a truncated circular cylinder delimited by plane-parallel end walls inclined with respect to the cylinder base. The cavity rotation axis coincides with the axis of the cylindrical surface. Libration-type forcing is introduced by harmonic modulation of the background rotation frequency. The sloping end walls break the axial symmetry of the liquid domain: the shape of the axial-radial cross sections varies from parallelogram to rectangle depending on the azimuthal angle. It is found that, regardless of the liquid response type (wave attractor or inertial modes), the transition from linear to nonlinear dynamics follows the scenario of triadic resonance instability. However, the time-averaged zonal flow responds differently to the primary wave instability. Inertial-mode instability generates a system of azimuthally periodic averaged vortices, whose frequency coincides with the subharmonic frequency of the triadic resonance. At high libration amplitudes, a low-frequency component appears in the azimuthal velocity spectrum, being associated with excitation of the retrograde system of vortices. The development of the weakly nonlinear regime of the wave attractor is accompanied by the instability of the viscous boundary layers—fine-scale pattern formation occurs close to the reflection zones of the attractor branches at the cylindrical sidewall. In the strongly nonlinear wave regime, coherent vortex structures are excited, performing azimuthal and radial drifts.