Complementarity and substitutability between products are essential concepts in retail and marketing. To achieve this, existing approaches take advantage of knowledge graphs to learn more evidence for inference. However, they often omit the knowledge that lies in the unstructured data. In this research, we concentrate on inferring complementary and substitutable products in e-commerce from mass structured and unstructured data. An improved knowledge-graph-based reasoning model has been proposed which cannot only derive related products but also provide interpretable paths to explain the relationship. The methodology employed in our study unfolds through several stages. First, a knowledge graph refining entities and relationships from data was constructed. Second, we developed a two-stage knowledge representation learning method to better represent the structured and unstructured knowledge based on TransE and SBERT. Then, the relationship inferring problem was converted into a path reasoning problem under the Markov decision process environment by learning a dynamic policy network. We also applied a soft pruning strategy and a modified reward function to improve the effectiveness of the policy network training. We demonstrate the effectiveness of the proposed method on standard Amazon datasets, and it gives about 5–15% relative improvement over the state-of-the-art models in terms of NDCG@10, Recall@10, Precision @10, and HR@10.