I describe a configurable machine‐learning framework to estimate a suite of continuous and categorical sedimentological properties from photographic imagery of sediment, and to exemplify how machine learning can be a powerful and flexible tool for automated quantitative and qualitative measurements from remotely sensed imagery. The model is tested on a dataset consisting of 409 images and associated detailed label data. The data are from a much wider sedimentological spectrum than previous optical granulometry studies, consisting of both well‐ and poorly sorted sediment, terrigenous, carbonate, and volcaniclastic sands and gravels and their mixtures, and grain sizes spanning over two orders of magnitude. I demonstrate the model framework by configuring it in several ways, to estimate two categories (describing grain shape and population, respectively) and nine numeric grain size percentiles in pixels from a single input image. Grain size is then recovered using the physical size of a pixel. Finally, I demonstrate that the model can be configured and trained to estimate equivalent sieve diameters directly from image features, without the need for area‐to‐mass conversion formulas and without even knowing the scale of one pixel. Thus it is the only optical granulometry method proposed to date that does not necessarily require image scaling. The flexibility of the model framework should facilitate numerous application in the spatiotemporal monitoring of the grain size distribution, shape, mineralogy and other quantities of interest of sedimentary deposits as they evolve, as well as other texture‐based proxies extracted from remotely sensed imagery. © 2019 John Wiley & Sons, Ltd.