The spectacular results provided by the second generation VLTI instruments GRAVITY and MATISSE on Active Galactic Nuclei trigger and justify a strong increase in the sensitivity limit of optical interferometers. A key component of such an upgrade is off-axis fringe tracking. To evaluate its potential and limitations, we describe and analyze its error budget including fringe sensing precision and temporal, angular and chromatic perturbations of the piston. The global tracking error is computed using standard seeing parameters for different sites, seeing conditions and telescope sizes for the current GRAVITY Fringe Tracker and a new concept of Hierarchical Fringe Tracker. Then, it is combined with a large catalog of guide star candidates from Gaia to produce sky coverage maps that give the probability to find a usable off-axis guide star in any part of the observable sky. These maps can be used to set the specifications of the system, check its sensitivity to seeing conditions and evaluate the feasibility of science programs. We check the availability of guide stars and the tracking accuracy for a large set of 15799 Quasars to confirm the feasibility of a large program on BLRs in the K band with the GRAVITY Fringe Tracker and show how it can be extended to the L, M and N bands. Another set of 331 well-characterized nearby AGNs shows the high potential of MATISSE for imaging and characterization of the dust torus in the N band under off-axis tracking on both UTs and ATs.