Context. In the near-future, dedicated telescopes will observe Earth-like exoplanets in reflected parent-starlight, allowing their physical characterization. Because of the huge distances, every exoplanet will remain an unresolved, single pixel, but temporal variations in the pixel's spectral flux contain information about the planet's surface and atmosphere.Aims. We test convolutional neural networks for retrieving a planet's rotation axis, surface and cloud map from simulated single-pixel observations of flux and polarization light curves. We investigate the influence of assuming that the reflection by the planets is Lambertian in the retrieval while in reality their reflection is bidirectional, and of including polarization in retrievals.Methods. We simulate observations along a planet's orbit using a radiative transfer algorithm that includes polarization and bidirectional reflection by vegetation, desert, oceans, water clouds, and Rayleigh scattering in six spectral bands from 400 to 800 nm, at various levels of photon noise. The surface-types and cloud patterns of the facets covering a model planet are based on probability distributions. Our networks are trained with simulated observations of millions of planets before retrieving maps of test planets.Results. The neural networks can constrain rotation axes with a mean squared error (MSE) as small as 0.0097, depending on the orbital inclination. On a bidirectionally reflecting planet, 92% of ocean facets and 85% of vegetation, desert, and cloud facets are correctly retrieved, in the absence of noise. With realistic amounts of noise, it should still be possible to retrieve the main map features with a dedicated telescope. Except for face-on orbits, a network trained with Lambertian reflecting planets, yields significant retrieval errors when given observations of bidirectionally reflecting planets, in particular, brightness artefacts around a planet's pole.