Asbestos can be present in a wide variety of heritage objects that are maintained in museums and heritage collections. Current research focuses on health risks, safety measures, and asbestos consolidation. However, there are additional challenges that remain understudied. Asbestos itself needs to be tackled as heritage, asking whether it is heritage, and if—in that case—the removal of asbestos affects the values and significance of an object. The collection of the former Belgian Asbestos and Rubber Factory (Belgium) is used in this contribution to illustrate why the heritage values of asbestos need to be considered, how asbestos challenges museums and heritage institutions, and why it needs to be theorized as difficult heritage. Subsequently, a methodology to value asbestos-containing heritage objects in a multi-voiced way is elaborated.