Advances in machine learning (ML) have enabled the development of next-generation prediction models for complex computational biology problems. These developments have spurred the use of interpretable machine learning (IML) to unveil fundamental biological insights through data-driven knowledge discovery. However, in general, standards and guidelines for IML usage in computational biology have not been well-characterized, representing a major gap toward fully realizing the potential of IML. Here, we introduce a workflow on the best practices for using IML methods to perform knowledge discovery which covers verification strategies that bridge data, prediction model, and explanation. We outline a workflow incorporating these verification strategies to increase an IML method's accountability, reliability, and generalizability. We contextualize our proposed workflow in a series of widely applicable computational biology problems. Together, we provide an extensive workflow with important principles for the appropriate use of IML in computational biology, paving the way for a better mechanistic understanding of ML models and advancing the ability to discover novel biological phenomena.