Mastering the Material Handling System (MHS) is still a crucial issue, according to Hellmann et al the costs of Material Handling activities ranges from 15–70% of total manufacturing costs depending on the kind of production [6]. The introduction of industry 4.0 technologies for Material Handling, is renewing the decisions to take in the design of MHS. The practices in MHS design and management are very diversified in industry and no consensus clearly exist on how efficiently designing such system. This paper discusses the need to understand the relation between the different aspects of Material Handling System design, and propose to identify the key challenges to be addressed. Through field studies that were conducted with five distinct companies and an analysis of the literature, five MHS aspects are examined; Material Handling Activities, Material Handling and automation, Material Handling Control System, Material Handling Equipment Selection, and Material Handling System. Based on the results of the analysis, a cross analysis is performed to identify the differences and the common patterns between the literature and the field study. As a result, research directions for new MHS design approach are proposed. It is composed of four main challenges; Material handling specifications, MHE selection, MHE deployment, and MHS analysis. Each challenge is linked to other potential subjects.