This paper deals with a mathematical modeling of flow stability of Newtonian and non-Newtonian fluids in the gap between two concentric cylinders, one of which rotates. A typical feature of the flow is the formation of a vortex flow, so-called Taylor vortices. Vortex structures are affected by the speed of the rotating cylinder and the physical properties of the fluids, i.e., viscosity and density. Analogy in terms of viscosity is assumed for non-Newtonian and magnetorheological fluids. Mathematical models of laminar, transient and turbulent flow with constant viscosity and viscosity as a function of the deformation gradient were formulated and numerically solved to analyze the stability of single-phase flow. To verify them, a physical experiment was performed for Newtonian fluids using visualizations of vortex structures—Taylor vortices. Based on the agreement of selected numerical and physical results, the experience was used for numerical simulations of non-Newtonian magnetorheological fluid flow.