We numerically study the quantum oscillations in superconducting vortex-mixed states with d-wave pairing. We show that in the parameter range of an underdoped cuprate superconductor, the commonly held assumption that the period is given by the underlying Fermi-surface area using the Onsager relation becomes invalid. Using this result, we conclude that the interpretation of the recent experimental data on YBCO as a signal of an underlying Fermi surface with four hole pockets created by a ͑ , ͒ folding cannot be ruled out. DOI: 10.1103/PhysRevB.79.180510 PACS number͑s͒: 74.25.Jb Recent experiments have shown quantum oscillations in underdoped YBCO samples in strong magnetic field ϳ45 T. [1][2][3][4][5] This has been interpreted as a signal of the underlying Fermi surface ͑FS͒. The period of the oscillation is large and implies that the FS has been reconstructed, probably, by some translational symmetry-breaking order in the ground state.6-9 However, for both materials that have been studied, the simplest construction, a ͑ , ͒ folding forming four hole pockets in the original Brillouin zone ͑BZ͒, would imply a pocket too small comparing to the nominal doping by about 25%. 1,3 This partly motivated a number of workers to interpret the data in terms of more complicated reconstruction such as incommensurate spin-density wave ͑SDW͒.
5,7Furthermore, the measured Hall effect is negative 10 and this led LeBoeuf et al. to propose that the quantum oscillations originate from electron pockets. Whether the negative Hall effect is due to flux flow 11 is currently a subject under debate. We believe that in these experiments, the samples are still in a vortex-mixed state. One evidence is the measured torque hysteresis, 5 which implies that vortices exist at least up to 45 T. Furthermore, the commonly quoted core size of 20 Å ͓see, for example, the extrapolation based on scanning tunneling microscopy ͑STM͒ measurement 12 as well as Nernst measurements 13 ͔ lead to an estimate of H c2 of 100 T. Therefore, to interpret the oscillations data, one has to understand the quantum oscillations in the mixed state.Up to now, all discussions assume that quantum oscillations in the mixed state maintain the same frequency as in the normal state and are given by the Onsager relation, which relates the frequency to the Fermi-surface area. This is true of all experiments performed up to date where it is possible to scan the magnetic field across H c2 . 14,15 However, there is no clear argument why the frequency should remain the same and previous theories predict a small shift. 16,17 We note that all previous experiments have been done on conventional s-wave low T c superconductors, with the possible exception of the organics which may be d-wave, and the high-T c cuprates may be in quite a different parameter regime. For example, the coherence length 0 is very short, on the order of 4 or 5 lattice constants, and is the consequence of a large energy gap ⌬ 0 . The number of Landau levels N is about 10 in the high-T c experiments, as opposed t...