Recently, as the types of imported food and the design of their packaging become more complex and diverse, digital recognition technologies such as barcodes, QR (quick response) codes, and OCR (optical character recognition) are attracting attention in order to quickly and easily check safety information (e.g., food ingredient information and recalls). However, consumers are still exposed to inaccurate and inconvenient situations because legacy technologies require dedicated terminals or include information other than safety information. In this paper, we propose a deep learning-based packaging recognition system which can easily and accurately determine food safety information with a single image captured through a smartphone camera. The detection algorithm learned a total of 100 kinds of product images and optimized YOLOv7 to secure an accuracy of over 95%. In addition, a new SUS (system usability scale)-based questionnaire was designed and conducted on 71 consumers to evaluate the usability of the system from the individual consumer’s perspective. The questionnaire consisted of three categories, namely convenience, accuracy, and usefulness, and each received a score of at least 77, which confirms that the proposed system has excellent overall usability. Moreover, in terms of task completion rate and task completion time, the proposed system is superior when it compared to existing QR code- or Internet-based recognition systems. These results demonstrate that the proposed system provides consumers with more convenient and accurate information while also confirming the sustainability of smart food consumption.