Value of Information (VoI) is a concept to assess the usefulness of information for a specific goal, and has in the last decade experienced a growing interest also for Wireless Sensor Network (WSN) applications and the Internet of Things (IoT). By making the value of information explicit in the form of VoI, WSN and IoT applications should be able to better assess which information to spend their constrained resources on. However, the definition of VoI is highly application-dependent, which has led to a fragmented understanding of VoI, and there is a lack of a comprehensive overview. In this structured review, we first categorize application use cases and examine what VoI is used for, and explore the different approaches to defining VoI. We then provide a well-structured and comprehensive discussion of the specific approaches used in the literature to determine VoI, together with examples of use cases. We categorize the different approaches to calculating VoI, describe their properties systematically and distinguish between observed VoI and expected VoI. We also discuss adaptive VoI approaches and point towards future directions within the field.