This paper evaluates VGG-16 and VGG-19 networks in performing semantic image segmentation of Malaysian meals. This is a preliminary investigation of using transfer learning models to recognize food objects in typical Malaysian meals. Most current works of food recognition system calculate the calories and nutritional content of a meal based on the food object recognition, regardless of the portion size. Our final aim is to develop a food recognition system that considers the portion size in calculating the calories and nutritional content. Therefore, semantic segmentation of the food objects in the meal is a very important stage. Our work also initiated the training datasets for Malaysian meals that will be made available to the public. Using a small training dataset and a basic configuration of the VGG network, our results show inconsistent findings of the performance of VGG-16 and VGG-19. These findings will serve as a fundamental guideline to improve the semantic segmentation of food images.