Multi-channel detection in single-molecule localization microscopy (SMLM) greatly increases information content for various biological applications. Here, we present globLoc, a graphics processing unit (GPU) based global fitting algorithm with flexible PSF modeling and parameter sharing, to extract maximum information from multi-channel single molecule data. We show, both in simulations and experiments, that global fitting can substantially improve the 3D localization precision for biplane and 4Pi SMLM and color assignment for ratiometric multicolor imaging.