Smart textiles have recently aroused tremendous interests over the world because of their broad applications in wearable electronics, such as human healthcare, human motion detection, and intelligent robotics. Sensors are the primary components of wearable and flexible electronics, which convert various signals and external stimuli into electrical signals. While traditional electronic sensors based on rigid silicon wafers can hardly conformably attach on the human body, textile materials including fabrics, yarns, and fibers afford promising alternatives due to their characteristics including light weight, flexibility, and breathability. Of fundamental importance are the needs for fabrics simultaneously having high electrical and mechanical performance. This article focused on the hierarchical design of the textile-based flexible sensor from a structure point of view. We first reviewed the selection of newly developed functional materials for textile-based sensors, including metals, conductive polymers, carbon nanomaterials, and other two-dimensional (2D) materials. Then, the hierarchical structure design principles on different levels from microscale to macroscale were discussed in detail. Special emphasis was placed on the microstructure control of fibers, configurational engineering of yarn, and pattern design of fabrics. Finally, the remaining challenges toward industrialization and commercialization that exist to date were presented.