Despite the promising properties of tea tree oil (TTO) as potential therapeutics for several superficial skin conditions, certain limitations such as physical instability and skin irritation have restricted its widespread use. This study focuses on developing a rationally designed lipid-based nanoformulation (TTO-LNF) in accordance with the US Food and Drug Administration standard using a well-recognized quality-by-design (QbD) approach. Using a mixture experimental design, TTO-LNF has been optimized with 5% TTO, 10% surfactant, 5% co-surfactant, and 80% water, which showed a 14.4 ± 4.4 nm droplet size and 0.03 ± 0.01 polydispersity index (PDI). To ease the topical administration, the TTO-LNF gel formulation was further developed using xanthan gum to achieve the desired viscosity and form a gel. The in vitro antibacterial tests of TTO-LNF showed promising inhibitory effects toward both Gram-negative and Gram-positive bacteria. In fact, a complete growth inhibition of S. epidermidis was observed when exposed to TTO-LNF and TTO-LNF gel for 24 h, showing better activity than antibiotic kanamycin (25 µg/mL). Additionally, the in vitro release study showed a sustained release profile with a 50% release in 24 h, which could be beneficial to reduce the toxicity and thereby improve the therapeutic efficacy for long-acting applications. Furthermore, the formulations were remarkably stable at 40 °C/75% Relative humidity (RH) for at least 4 weeks. Therefore, this study presents a promising strategy to develop a biocompatible and stable formulation that can be used for the topical treatment of skin infections.