Network data has become widespread, larger, and more complex over the years. Traditional network data is dyadic, capturing the relations among pairs of entities. With the need to model interactions among more than two entities, significant research has focused on higher-order networks and ways to represent, analyze, and learn from them. There are two main directions to studying higher-order networks. One direction has focused on capturing higher-order patterns in traditional (dyadic) graphs by changing the basic unit of study from nodes to small frequently observed subgraphs, called motifs. As most existing network data comes in the form of pairwise dyadic relationships, studying higher-order structures within such graphs may uncover new insights. The second direction aims to directly model higher-order interactions using new and more complex representations such as simplicial complexes or hypergraphs. Some of these models have long been proposed, but improvements in computational power and the advent of new computational techniques have increased their popularity. Our goal in this paper is to provide a succinct yet comprehensive summary of the advanced higher-order network analysis techniques. We provide a systematic review of its foundations and algorithms, along with use cases and applications of higher-order networks in various scientific domains.Higher-Order Networks Representation and Learning: A Survey A PREPRINT detailed mathematical representations of them, we briefly describe their differences using concrete examples involving three individuals A, B, and C (21):1. Network Motifs: A, B, and C have contact information of each other in their contact list. They form a triangle (an example of a motif ).2. Simplicial Complexes: A, B, and C are in the same class in high school. Any subset of {A, B, C} indicates a classmate relationship. A, B, and C form a simplex, a basic unit of a simplicial complex.3. Hypergraphs: A, B, and C publish a paper together. We create a new hyperedge representing the collection of all authors of this paper, i.e., hyperedge {A, B, C}. A, B, and C form a hypergraph with a single hyperedge. Note that {A, B},{A, C} and {B, C} are not included as hyperedges in the hypergraph.