The coffee supply chain is characterized by a complex network with many critical and unsustainable points producing a huge amount of waste products. Among these, coffee silverskin (CS), the only by-product of the coffee roasting phase, has an interesting chemical profile that suggests potential use as a food ingredient. However, few data on its safety are available. For this reason, the purpose of the study was to assess the occurrence of chemical and biological contaminants in CS, and the resulting risk due to its potential consumption. Essential, toxic, and rare earth elements, polycyclic aromatic hydrocarbons (PAHs), process contaminants, ochratoxin A (OTA), and pesticides residues were analyzed in three classes of samples (Coffea arabica CS, Coffea robusta CS, and their blend). Furthermore, total mesophilic bacteria count (TMBC) at 30 °C, Enterobacteriaceae, yeasts, and molds was evaluated. The risk assessment was based upon the hazard index (HI) and lifetime cancer risk (LTCR). In all varieties and blends, rare earth elements, pesticides, process contaminants, OTA, and PAHs were not detected except for chrysene, phenanthrene, and fluoranthene, which were reported at low concentrations only in the arabica CS sample. Among essential and toxic elements, As was usually the most representative in all samples. Microorganisms reported a low load, although arabica and robusta CS showed lower contamination than mixed CS. Instead, the risk assessment based on the potential consumption of CS as a food ingredient did not show either non-carcinogenic or carcinogenic risk. Overall, this study provides adequate evidence to support the safety of this by-product for its potential use in functional foods.