Numerical simulation of ultrasound images can facilitate the training of sonographers. An efficient and realistic model for the simulation of ultrasonic speckle is the convolution of the ultrasound point-spread function with a distribution of point scatterers. Nevertheless, for a given arbitrary tissue type, a scatterer map that would generate a realistic appearance of that tissue is not known a priori. In this paper, we introduce a principled approach to estimate (reconstruct) such a scatterer map from images, by solving the inverse-problem of ultrasound speckle formation, such that images from arbitrary view angles and transducer settings can be generated from those scatterer maps later in simulations. Robust reconstructions are achieved by using multiple measurements of the same tissue with different viewing parameters. For this purpose, a novel use of beam-steering to rapidly and conveniently acquire multiple images of the same scene is proposed. We demonstrate in numerical and physical phantoms and images that the appearance of synthesized images closely match real images for a range of viewing parameters and probe settings. We also present a scene editing scenario exploiting these scatterer representations to create realistic images of augmented anatomy.