The Monaco Monte Carlo treatment planning system uses three‐beam model components to achieve accuracy in dose calculation. These components include a virtual source model (VSM), transmission probability filters (TPFs), and an x‐ray voxel Monte Carlo (XVMC) engine to calculate the dose in the patient. The aim of this study was to assess the TPF component of the Monaco TPS and optimize the TPF parameters using measurements from an Elekta linear accelerator with an Agility™ multileaf collimator (MLC). The optimization began with all TPF parameters set to their default value. The function of each TPF parameter was characterized and a value was selected that best replicated measurements with the Agility™ MLC. Both vendor provided fields and a set of additional test fields were used to create a rigorous systematic process, which can be used to optimize the TPF parameters. It was found that adjustment of the TPF parameters based on this process resulted in improved point dose measurements and improved 3D gamma analysis pass rates with Octavius 4D. All plans calculated with the optimized beam model had a gamma pass rate of > 95% using criteria of 2% global dose/2 mm distance‐to‐agreement, while some plans calculated with the default beam model had pass rates as low as 88.4%. For measured point doses, the improvement was particularly noticeable in the low‐dose regions of the clinical plans. In these regions, the average difference from the planned dose reduced from 4.4 ± 4.5% to 0.9 ± 2.7% with a coverage factor (k = 2) using the optimized beam model. A step‐by‐step optimization guide is provided at the end of this study to assist in the optimization of the TPF parameters in the Monaco TPS. Although it is possible to achieve good clinical results by randomly selecting TPF parameter values, it is recommended that the optimization process outlined in this study is followed so that the transmission through the TPF is characterized appropriately.