This paper presents a prospective view of how much calories is consumed by a person before and after a meal. From the calorie intake project, the management gets to know whether the person consumed the necessary amount of calorie. Although the hospital diet can provide adequate energy and nutrients, many patients may not consume sufficient food to meet their needs. The estimated energy intake of about one-third of patients was very low, and vitamin C, calcium and zinc intakes were also of concern. At least one-third of patients arriving at the hospital are malnourished. This can mean a patient’s diet does not provide the right amount or appropriate ratio of calories, macronutrients, or micronutrients to promote adequate healing. If this continuous then it may be a risk to the patient’s health and may not be a good sign. In this project, we proposed a deep learning system to monitor the nutrient intake for hospitalized patients. A Camera will scan the amount of food given to the patient, when the patient finishes the meal again the camera will again scan the amount of food the patient has taken. It will also note down the number of calories the patient has intake and the amount of calories the patient has left. This will help hospitalized patient to be healthy and will also help hospital staffs to monitor with ease.