Although there have been extensive research and progress on the discovery of anticancer drug over the years, the application of these drugs as stand-alone therapy has been limited by their off-target toxicities, poor pharmacokinetic properties, and low therapeutic index. Targeted drug delivery, especially drug conjugate, has been recognized as a technology that can bring forth a new generation of therapeutics with improved efficacy and reduced side effects for cancer treatment. The linker in a drug conjugate is of essential importance because it impacts the circulation time of the conjugate and the release of the drug for full activity at the target site. Recently, the light-triggered linker has attracted a lot of attention due to its spatiotemporal controllability and attractive prospects of improving the overall pharmacokinetics of the conjugate. In this paper, the latest developments of UV- and IR-triggered linkers and their application and potential in drug conjugate development are reviewed. Some of the most-well-researched photoresponsive structural moieties, such as UV-triggered coumarin, ortho-nitrobenzyl group (ONB), thioacetal ortho-nitrobenzaldehyde (TNB), photocaged C40-oxidized abasic site (PC4AP), and IR-triggered cyanine and BODIPY, are included for discussion. These photoremovable linkers show better physical and chemical stabilities and can undergo rapid cleavage upon irradiation. Very importantly, the drug conjugates containing these linkers exhibit reduced off-target toxicity and overall better pharmacokinetic properties. The progress on photoactive antibody–drug conjugates, such as antibody–drug conjugates (ADC) and antibody–photoabsorber conjugate (APC), as precision medicine in clinical cancer treatment is highlighted.