Vesicles are locally-inextensible fluid membranes that can sustain bending. In this paper, we extend "A numerical method for simulating the dynamics of 3D axisymmetric vesicles suspended in viscous flows", Veerapaneni et al. Journal of Computational Physics, 228(19), 2009 to general non-axisymmetric vesicle flows in three dimensions.Although the main components of the algorithm are similar in spirit to the axisymmetric case (spectral approximation in space, semi-implicit time-stepping scheme), important new elements need to be introduced for a full 3D method. In particular, spatial quantities are discretized using spherical harmonics, and quadrature rules for singular surface integrals need to be adapted to this case; an algorithm for surface reparameterization is neeed to ensure sufficient of the timestepping scheme, and spectral filtering is introduced to maintain reasonable accuracy while minimizing computational costs. To characterize the stability of the scheme and to construct preconditioners for the iterative linear system solvers used in the semi-implicit time-stepping scheme, we perform a spectral analysis of the evolution operator on the unit sphere.By introducing these algorithmic components, we obtain a time-stepping scheme that, in our numerical experiments, is unconditionally stable. We present results to analyze the cost and convergence rates of the overall scheme. To illustrate the applicability of the new method, we consider a few vesicle-flow interaction problems: a single vesicle in relaxation, sedimentation, shear flows, and many-vesicle flows.