Palm oil clinker (POC) is a waste material generated in large quantities from the palm oil industry. POC, when crushed, possesses the potential to serve as an aggregate for concrete production. Experimental investigation on the engineering properties of concrete incorporating POC as aggregate and filler material was carried out in this study. POC was partially and fully used to replace natural coarse aggregate. The volumetric replacements used were 0, 20%, 40%, 60%, 80%, and 100%. POC, being highly porous, negatively affected the fresh and hardened concrete properties. Therefore, the particle-packing (PP) method was adopted to measure the surface and inner voids of POC coarse aggregate in the mixtures at different substitution levels. In order to enhance the engineering properties of the POC concrete, palm oil clinker powder (POCP) was used as a filler material to fill up and coat the surface voids of POC coarse, while the rest of the mix constituents were left as the same. Fresh and hardened properties of the POC concrete with and without coating were determined, and the results were compared with the control concrete. The results revealed that coating the surface voids of POC coarse with POCP significantly improved the engineering properties as well as the durability performance of the POC concrete. Furthermore, using POC as an aggregate and filler material may reduce the continuous exploitation of aggregates from primary sources. Also, this approach offers an environmental friendly solution to the ongoing waste problems associated with palm oil waste material.