Reflected light photometry of terrestrial exoplanets could reveal the presence of oceans and continents, hence placing direct constraints on the current and long-term habitability of these worlds. Inferring the albedo map of a planet from its observed light curve is challenging because different maps may yield indistinguishable light curves. This degeneracy is aggravated by changing clouds. It has previously been suggested that disk-integrated photometry spanning multiple days could be combined to obtain a cloud-free surface map of an exoplanet. We demonstrate this technique as part of a Bayesian retrieval by simultaneously fitting for the fixed surface map of a planet and the time-variable overlying clouds. We test this approach on synthetic data then apply it to real disk-integrated observations of the Earth. We find that eight days of continuous synthetic observations are sufficient to reconstruct a faithful low resolution surface albedo map, without needing to make assumptions about cloud physics. For lightcurves with negligible photometric uncertainties, the minimal top-of-atmosphere albedo at a location is a good estimate of its surface albedo. When applied to observations from the Earth Polychromating Imaging Camera aboard the DSCOVR spacecraft, our approach removes only a small fraction of clouds. We attribute this difficulty to the full-phase geometry of observations combined with the short correlation length for Earth clouds. For exoplanets with Earth-like climatology, it may be hard to do much better than a cloud-averaged map. We surmise that cloud removal will be most successful for exoplanets imaged near quarter phase that harbour large cloud systems.