We investigate the effects of finite size corrections on the overlap probabilities in the Generalized Random Energy Model (GREM) in two situations where replica symmetry is broken in the thermodynamic limit. Our calculations do not use replicas, but shed some light on what the replica method should give for finite size corrections. In the gradual freezing situation, which is known to exhibit full replica symmetry breaking, we show that the finite size corrections lead to a modification of the simple relations between the sample averages of the overlaps Y k between k configurations predicted by replica theory. This can be interpreted as fluctuations in the replica block size with a negative variance. The mechanism is similar to the one we found recently in the random energy model [1]. We also consider a simultaneous freezing situation, which is known to exhibit one step replica symmetry breaking. We show that finite size corrections lead to full replica symmetry breaking and give a more complete derivation of the results presented in [2] for the directed polymer on a tree.