Chern numbers can be calculated within a frame of vortex fields related to phase conventions of a wave function. In a band protected by gaps the Chern number is equivalent to the total number of flux carrying vortices. In the presence of topological defects like Dirac cones this method becomes problematic, in particular if they lack a well-defined winding number. We develop a scheme to include topological defects into the vortex field frame. A winding number is determined by the behavior of the phase in reciprocal space when encircling the defect's contact point. To address the possible lack of a winding number we utilize a more general concept of winding vectors. We demonstrate the usefulness of this ansatz on Dirac cones generated from bands of the Hofstadter model.