In the present article, the cell model (or self-consistent scheme) is used to derive constitutive equations for rod suspensions in non-Newtonian viscous matrices such as power-law, Ellis and Carreau fluids. It is found that the shear-thinning character of the matrix influences considerably the rod contribution to the stress tensor, but has no impact on the rod orientation dynamics: the same microstructure evolution as the one encountered in Newtonian fluids is obtained. The rod suspension behaves differently than the unfilled matrix in the sense that, depending on rod orientation, the onset of shear thinning in the composite occurs at lower or higher shear rates. Our analysis also provides a semi-analytical model for rod suspensions in an Ellis fluid, which appears to be suitable for predicting a Newtonian plateau at low shear rates and a shear-thinning behaviour at high shear rates. In addition, the model predictions are in good agreement with the shear viscosity measurements of glass-fibre-filled polystyrene melts (Chan et al., J. Rheol., vol. 22 (5), 1978, pp. 507–524), demonstrating its ability to describe the rheological behaviour of such polymer composites. Finally, the proposed approach is extended to a Carreau fluid although its solution requires the numerical solution of a set of partial differential equations.