This work provides a systematic evaluation of how survey design and computer processing choices (such as the software used or the workflow/parameters chosen) influence unmanned aerial vehicle (UAV)-based photogrammetry retrieval of tree diameter at breast height (DBH), an important 3D structural parameter in forest inventory and biomass estimation. The study areas were an agricultural field located in the province of Málaga, Spain, where a small group of olive trees was chosen for the UAV surveys, and an open woodland area in the outskirts of Sofia, the capital of Bulgaria, where a 10 ha area grove, composed mainly of birch trees, was overflown. A DJI Phantom 4 Pro quadcopter UAV was used for the image acquisition. We applied structure from motion (SfM) to generate 3D point clouds of individual trees, using Agisoft and Pix4D software packages. The estimation of DBH in the point clouds was made using a RANSAC-based circle fitting tool from the TreeLS R package. All trees modeled had their DBH tape-measured on the ground for accuracy assessment. In the first study site, we executed many diversely designed flights, to identify which parameters (flying altitude, camera tilt, and processing method) gave us the most accurate DBH estimations; then, the resulting best settings configuration was used to assess the replicability of the method in the forested area in Bulgaria. The best configuration tested (flight altitudes of about 25 m above tree canopies, camera tilt 60°, forward and side overlaps of 90%, Agisoft ultrahigh processing) resulted in root mean square errors (RMSEs; %) of below 5% of the tree diameters in the first site and below 12.5% in the forested area. We demonstrate that, when carefully designed methodologies are used, SfM can measure the DBH of single trees with very good accuracy, and to our knowledge, the results presented here are the best achieved so far using (above-canopy) UAV-based photogrammetry.