Aims. We explore potential mechanisms to produce extremely high Lyα/HeII flux ratios, or to enhance the observed number of Lyα photons per incident ionizing photon, in extended AGN-photoionized nebulae at high-redshift. Methods. We compute models to simulate, in the low density regime, photoionization of interstellar gas by the radiation field of a luminous AGN. We explore the impact of ionization parameter, gas metallicity, ionizing spectrum, electron energy distribution, and cloud viewing angle on the relative fluxes of Lyα, HeII and other lines, and on the observed number of Lyα photons per incident ionizing photon. We compare our model results with recent observations of quasar Lyα halos at z∼3.5. Results. Low ionization parameter, a relatively soft or filtered ionizing spectrum, low gas metallicity, κ-distributed electron energies, or reflection of Lyα photons by neutral hydrogen can all result in significantly enhanced Lyα relative to other lines (≥ 10 %), with log Lyα/HeII reaching values of up to 4.6. In the cases of low gas metallicity, reflection by HI, or a hard or filtered ionizing spectrum, the observed number of Lyα photons per incident ionizing photon is itself significantly enhanced above the nominal Case B value of 0.66 due to collisional excitation, reaching values as high as 5.3 in an 'extreme case' model which combines several of these effects. We find that at low gas metallicity (e.g. Z/Z ⊙ =0.1) the production of Lyα photons is predominantly via collisional excitation rather than by recombination. In addition, we find that the collisional excitation of Lyα becomes much more efficient if the ionizing continuum spectrum has been pre-filtered through an optically thin screen of gas closer to the AGN (e.g., by a wide-angle, feedbackdriven outflow.) We also show that the Lyα and HeII emission line ratios of the z∼3.5 quasars studied by Borisova et al. (2016) are consistent with AGN-photoionization of gas with moderate to low metallicity and/or low ionization parameter, without requiring exotic ionization/excitation mechanisms such as strong line-transfer effects. In addition, we present a set of UV-optical diagnostic diagrams to distinguish between photoionization by Pop III stars and photoionization by an AGN.