Due to the widespread proliferation of today's Internet of Things (IoT), a system designer needs the IoT system and software design patterns to assist in designing scalable and replicable solutions. Patterns are encapsulations of reusable common problems and solutions under specific contexts. Many IoT patterns have been published, such as IoT design patterns and IoT architecture patterns to document the successes (and failures) in IoT systems and software development. However, because these patterns are not well classified, their adoption does not live up to their potential. To understand the reasons, we conducted a systematic literature review. From the 32 identified papers, 143 IoT architecture and design patterns were extracted. We analyzed these patterns according to several characteristics and outlined directions for improvements when publishing and adopting IoT patterns. Of the extracted patterns, 57% are non-IoT patterns, suggesting that IoT systems and software are often designed via conventional architecture and design patterns that are not specific to IoT design. Although most IoT design patterns are applicable to any domain, IoT architecture patterns tend to be domain specific, implying that the unique nature of IoT adoption in specific domains appears at the architecture level. As more domains adopt IoT, the number of domain-specific IoT design patterns should increase. In terms of quality attributes, many IoT patterns address compatibility, security, and maintainability.