We propose a general theoretical framework, using two layers of ancilla qubits, for a continuous transition between a Fermi liquid with a large Fermi surface, and a pseudogap metal with a small Fermi surface of electron-like quasiparticles. The pseudogap metal can be a magnetically ordered metal, or a fractionalized Fermi liquid (FL*) without magnetic order. A critical 'ghost' Fermi surface emerges (alongside the large electron Fermi surface) at the quantum critical point, with the ghost fermions carrying neither spin nor charge, but minimally coupled to (U (1) × U (1))/Z 2 or (SU (2) × U (1))/Z 2 gauge fields. Away from the critical point on the pseudogap side, the ghost Fermi surface absorbs part of the large electron Fermi surface, and leads to a jump in the Hall co-efficient. We also find an example of an "unnecessary quantum critical point" between a metal with spin density order, and a metal with local moment magnetic order. The ghost fermions contribute a T ln(1/T ) specific heat at temperature T at the critical point, and could also be detected in other thermal probes. We relate our results to the phases of correlated electron compounds.