Inspired by non-abelian vortex anyons in spinor Bose-Einstein condensates, we consider the quantum double D(Q8) anyon model as a platform to carry out a particular instance of Shor's factorization algorithm. We provide the excitation spectrum, the fusion rules, and the braid group representation for this model, and design a circuit architecture that facilitates the computation. All necessary quantum gates, less one, can be compiled exactly for this hybrid topological quantum computer, and to achieve universality the last operation can be implemented in a non-topological fashion. To analyse the effect of decoherence on the computation, a noise model based on stochastic unitary rotations is considered. The computational potential of this quantum double anyon model is similar to that of the Majorana fermion based Ising anyon model, offering a complementary future platform for topological quantum computation.