The objective of this work is to determine the effect of the rising motion on the dynamics of inertial shape oscillations of drops and bubbles. We have carried out axisymmetric direct numerical simulations of an ascending drop (or bubble) using a level-set method. The drop is initially elongated in the vertical direction and therefore performs shape oscillations. The analysis is based on the decomposition of the interface into spherical harmonics, the time evolutions of which are processed to obtain the frequency and the damping rate of the oscillations. As the drop accelerates, its shape flattens and oscillations no longer take place around a spherical equilibrium shape. This causes the eigenmode of oscillations to change, which results in the appearance of spherical harmonics of high order that all oscillate at the same frequency. For both drops and bubbles, the frequency, which remains controlled by the potential flow, slightly decreases with the rising velocity. The damping rate of drops, which is controlled by the dissipation within boundary layers at the interface, strongly increases with the rising velocity. At terminal velocity, the damping rate of bubbles, which results from the dissipation by the potential flow associated with the oscillating motion, remains close to that of a non-rising bubble. During the transient, the rate of deformation of the equilibrium shape of bubbles can be comparable to the oscillation frequency, which causes complex evolutions of the shape. These results extend the description of shape oscillations to common situations where gravity plays a role. In particular, the present conclusions are useful to interpret experimental results where the effect of the rising motion is often combined with that of surfactant.