Toll-like receptors (TLRs), also known as pattern recognition receptors, respond to exogenous pathogens and to intrinsic danger signals released from damaged cells and tissues. The tetrapyrrole heme has been suggested to be an agonist for TLR4, the receptor for the pro-inflammatory bacterial component lipopolysaccharide (LPS), synonymous with endotoxin. Heme is a double-edged sword with contradictory functions. On the one hand, it has vital cellular functions as the prosthetic group of hemoproteins including hemoglobin, myoglobin, and cytochromes. On the other hand, if released from destabilized hemoproteins, non-protein bound or “free” heme can have pro-oxidant and pro-inflammatory effects, the mechanisms of which are not fully understood. In this review, the complex interactions between heme and TLR4 are discussed with a particular focus on the role of heme-binding serum proteins in handling extracellular heme and its impact on TLR4 signaling. Moreover, the role of heme as a direct and indirect trigger of TLR4 activation and species-specific differences in the regulation of heme-dependent TLR4 signaling are highlighted.