Objective: Non-invasive brain stimulation (NIBS) methodologies, such as transcranial electric (tES) and magnetic stimulation are increasingly employed for therapeutic, diagnostic, or research purposes. The concurrent presence of active or passive implants can pose safety risks, affect the NIBS delivery, or generate confounding signals. A systematic investigation is required to understand the interaction mechanisms, quantify exposure, assess safety, and establish guidance for NIBS applications. Approach: We used measurements, simplified generic, and detailed anatomical modeling to: (i) systematically analyze exposure conditions with passive and active implants, considering local field enhancement, exposure dosimetry, tissue heating and neuromodulation, capacitive lead current injection, low-impedance pathways between electrode contacts, and insulation damage; (ii) identify safety metrics and efficient prediction strategies; (iii) quantify these metrics in relevant exposure cases and (iv) identify worst case conditions. Various aspects including implant design, positioning, scar tissue formation, anisotropy, and frequency were investigated. Results: At typical tES frequencies, local enhancement of dosimetric exposure quantities can reach up to one order of magnitude for DBS and SEEG implants (more for elongated passive implants), potentially resulting in unwanted neuromodulation that can confound results but is still 2-3 orders of magnitude lower than active DBS. Under worst-case conditions, capacitive current injection in the lead of active implants can produce local exposures of similar magnitude as the passive field enhancement, while capacitive pathways between contacts are negligible. Above 10\,kHz, applied current magnitudes increase, necessitating consideration of tissue heating. Furthermore, capacitive effects become more prominent, leading to current injection that can reach DBS-like levels. Adverse effects from abandoned/damaged leads in direct electrode vicinity cannot be excluded. Significance: Safety related concerns of tES application in the presence of implants are systematically identified and explored, resulting in specific and quantitative guidance and establishing a basis for safety standards. Furthermore, several methods for reducing risks are suggested.