Infectious pancreatic necrosis (IPN) is a disease of great concern in aquaculture, mainly among salmonid farmers, since losses in salmonid fish—mostly very young rainbow trout (Salmo gairdnery) fry and Atlantic salmon (Salmo salar) post-smolt—frequently reach 80–90% of stocks. The virus causing the typical signs of the IPN disease in salmonids, named infectious pancreatic necrosis virus (IPNV), has also been isolated from other fish species either suffering related diseases (then named IPNV-like virus) or asymptomatic; the general term aquabirnavirus is used to encompass all these viruses. Aquabirnaviruses are non-enveloped, icosahedral bisegmented dsRNA viruses, whose genome codifies five viral proteins, three of which are structural, and one of them is an RNA-dependent RNA polymerase. Due to the great importance of the disease, there have been great efforts to find a way to predict the level of virulence of IPNV isolates. The viral genome and proteins have been the main focus of research. However, to date such a reliable magic marker has not been discovered. This review describes the processes followed for decades in the attempts to discover the viral determinants of virulence, and to help the reader understand how viral components can be involved in virulence modulation in vitro and in vivo. There is also a brief description of the disease, of host defenses, and of the molecular structure and function of the virus and its viral components.