Objective: Intensity-modulated radiotherapy (IMRT) is widely used in clinical radiotherapy, treating varying malignancies with conformal doses. As the test field for clinical translation, preclinical small animal experiments need to mimic the human radiotherapy condition, including IMRT. However, small animal IMRT is a systematic challenge due to the lack of corresponding hardware and software for miniaturized targets. Approach: The sparse orthogonal collimators (SOC) based on the direct rectangular aperture optimization (RAO) substantially simplified the hardware for miniaturization. This study investigates and evaluates a significantly improved RAO algorithm for complex mouse irradiation using SOC. Because the Kronecker product representation of the rectangular aperture is the main limitation of the computational performance, we reformulated matrix multiplication in the data fidelity term using multiplication with small matrices instead of the Kronecker product of the dose loading matrices. Solving the optimization problem was further accelerated using the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA). Main results: Four mouse cases, including a liver, a brain tumor, a concave U-target, and a complex total marrow irradiation (TMI) case, were included in this study with manually delineated targets and OARs. Seven coplanar-field SOC IMRT plans were compared with IMRT plans using an idealistic multi-leaf collimator (MLC). For the first three cases with simpler and smaller targets, the differences between SOC plans and MLC plans in the planning target volumes (PTV) statistics are within 1%. For the TMI case, the SOC plans are superior in reducing hot spots (also termed Dmax) of PTV, kidneys, lungs, heart, and bowel by 20.5%, 31.5%, 24.67%, 20.13%, and 17.78%, respectively. On average, in four cases in this study, the SOC plan conformity is comparable to that of the MLC’s with lightly increased R50 and Integral Dose by 2.23% and 2.78%. Significance: The significantly improved SOC optimization method allows fast plan creation in under 1 minute for smaller targets and makes complex TMI planning feasible while achieving comparable dosimetry to hypothetical IMRT with a theoretical MLC.