Internet of Things (IoT) is considered the upcoming industrial and academic revolution in the technological world having billions of things and devices connected to the Internet. These connected devices are heterogeneous. They have different standards and technologies which communicate through different protocols. Therefore, the implementation of IoT on a large scale is difficult due to these heterogeneity challenges. This motivated us to overcome the scaling problem of IoT by identifying the challenges from the literature and providing solutions. This study is based on the identification of the heterogeneous challenges with solutions via a systematic literature review. A total of 81 primary sources were selected. After extracting and synthesizing the data, we identified 14 different IoT heterogeneity challenges. Some of the identified challenges are “heterogeneity of devices,” “heterogeneity in formats of data,” “heterogeneity in communication,” and “interoperability issue due to heterogeneity.” The identified challenges have been analyzed from digital libraries and timeframe perspectives. Furthermore, we have found a total of 81 solutions for those challenges, with at least 5 unique solutions for each challenge. In the future, we will categorize the challenges and prioritize the solutions by using a multi-criteria decision-making problem.