Endoscopic retrograde cholangiopancreatography (ERCP) is an advanced technique using a side-viewing upper endoscope to diagnose and treat pancreaticobiliary diseases. ERCP is generally considered a safe procedure; however, it is associated with risks of certain complications such as pancreatitis, bowel perforation, bleeding, and infections. Very rarely, ERCP can result in abscess formation in different organs, such as the pancreas, liver, and intestines. Physicians should be vigilant for rare post-ERCP complications such as clinically significant bacteremia and hepatic abscess, especially in high-risk populations, as if left untreated, they can result in significant morbidity and mortality. We present an interesting and rare case of an 80-year-old patient who presented with nausea, vomiting, and abdominal pain post-ERCP and was found to have a polymicrobial bloodstream infection and a hepatic abscess. The patient was treated with medical therapy alone, with an appropriate clinical response.