Ammonium sulfate is a giant inverse barocaloric material that is cheaply and commercially available. Exploiting its potential for cooling applications requires an understanding of the mechanism driving the entropy change. Here we report an investigation by inelastic neutron scattering and density functional theory of the phonons under working conditions of temperature and pressure. We find excellent agreement between the experimental and calculated results. The ammonium librational modes that are crucial to the entropy change are identifiable by their negative Grüneisen parameter. Our results connect the differences in structure across the phase transition to those in the atomic dynamics, suggesting a route towards designing new caloric materials.