Trustworthiness and product traceability are essential factors in the apparel industry 4.0 for establishing successful business relationships among stakeholders such as customers, manufacturers, suppliers, and consumers. Each stakeholder has implemented different technology-based systems to record and track product transactions. However, these systems work in silos, and there is no intra-system communication, leading to a lack of complete supply chain traceability for all apparel stakeholders. Moreover, apparel stakeholders are reluctant to share their business information with business competitors; thus, they involve third-party auditors to ensure the quality of the final product. Furthermore, the apparel manufacturing industry faces challenges with counterfeit products, making it difficult for consumers to determine the authenticity of the products. Therefore, in this paper, a trustworthy apparel product traceability framework called ChainApparel is developed using the Internet of Things (IoT) and blockchain to address these challenges of authenticity and traceability of apparel products. Specifically, multiple smart contracts are designed and developed for registration, process execution, audit, fault, and product traceability to authorize, validate, and trace every business transaction among the apparel stakeholders. Further, the real-time performance analysis of ChainApparel is carried out regarding transaction throughput and latency by deploying the compute nodes at different geographical locations using Hyperledger Fabric. The results conclude that ChainApparel accomplished significant performance under diverse workloads while ensuring complete traceability along the complex supply chain of the apparel industry. Thus, the ChainApparel framework helps make the apparel product more trustworthy and transparent in the market while safeguarding trust among the industry stakeholders.