Sesame seed hull is the major by-product of sesame seed processing and is rich in polysaccharides. In this work, sesame hull polysaccharides (SHP) were extracted by ultrasound-assisted alkali extraction methods with a yield of 6.49%. Three purified polysaccharide fractions were obtained after decolorization, deproteinization, and column chromatography. Then, their main composition and antioxidant activity were investigated. The dominant fraction was SHP-2 with a yield of 3.78%. It was composed of galacturonic acid (51.3%), glucuronic acid (13.8%), rhamnose (8.9%), glucose (8.4%), and others. The linkage types of SHP-2 have the α-D-GalpA-(1,4)-linked, α-D-GlcpA-(1,2)-linked, β-T-D-Rhap-linked, β-D-Glcp-(1,6)-linked, β-T-D-Galp-linked, α-L-Xylp-(1,4)-linked, α-L-Araf-(1,3,5)-linked, and β-D-Manp-(1,4)-linked. This study might provide some useful basic data for developing applications for sesame seed hull polysaccharides in the food and pharmaceutical industries.