There has been strong demand for the development of an accurate but simple method to assess the freshness of food. In this study, we demonstrated a system to determine food freshness by analyzing the spectral response from a portable visible/near-infrared (VIS/NIR) spectrometer using the Convolutional Neural Network (CNN)-based machine learning algorithm. Spectral response data from salmon, tuna, and beef incubated at 25 °C were obtained every minute for 30 h and then categorized into three states of “fresh”, “likely spoiled”, and “spoiled” based on time and pH. Using the obtained spectral data, a CNN-based machine learning algorithm was built to evaluate the freshness of experimental objects. In addition, a CNN-based machine learning algorithm with a shift-invariant feature can minimize the effect of the variation caused using multiple devices in a real environment. The accuracy of the obtained machine learning model based on the spectral data in predicting the freshness was approximately 85% for salmon, 88% for tuna, and 92% for beef. Therefore, our study demonstrates the practicality of a portable spectrometer in food freshness assessment.