The combined effect of space charge and nonlinear resonance on beam loss and emittance was measured in a benchmarking experiment over a 1.2 s long flat bottom at 1.4 GeV kinetic energy in the presence of a single controllable octupole. By lowering the working point towards the resonance, a gradual transition from a loss-free core emittance blowup to a regime dominated by continuous loss was found. We compare the observation with 3D simulations based on a new analytical space charge model and obtain good agreement in the emittance blowup regime. Our explanation is in terms of the synchrotron oscillation, which causes a periodic tune modulation due to space charge, and leads to trapping and detrapping on the resonance islands. For working points very close to the resonance this induces a beam halo with large radius. The underlying dynamics is studied in detail, and it is claimed that the predicted halo in conjunction with a reduced dynamic aperture for the real machine lattice is the source of the loss observed in the experiment.