In this study, we examine the third-order fractional nonlinear Schrödinger equation (FNLSE) in $$(1+1)$$-dimensional, by employing the analytical methodology of the new extended direct algebraic method (NEDAM) alongside optical soliton solutions. In order to better understand high-order nonlinear wave behaviors in such systems, the researched model captures the physical and mathematical properties of nonlinear dispersive waves, with applications in plasma physics and optics. With the aid of above mentioned approach, we rigorously assess the novel optical soliton solutions in the form of dark, bright–dark, dark–bright, periodic, singular, rational, mixed trigonometric and hyperbolic forms. Additionally, stability assessments using conserved quantities, such as Hamiltonian property, and consistency checks were used to validate the solutions. The dynamic structure of the governing model is further examined using chaos, bifurcation, and sensitivity analysis. With the appropriate parameter values, 2D, 3D, and contour plots can all be utilized to graphically show the data. This work advances our knowledge of nonlinear wave propagation in Bose–Einstein condensates, ultrafast fibre optics, and plasma physics, among other areas with higher-order chromatic effects.