A novel videophone system for the elderly-care application is proposed. Based on the detailed analysis of the elderly’s physical and psychological characteristics, a TV-based caring videophone system for the elderly is developed: an embedded multimedia device is designed to implement the interactive video and audio processing and IP-based communication, in which TV is adopted as the display terminal to achieve a low-cost but high-quality service. Considering the user’s convenience, many personalized designs, such as photo-based address book, photo-click-dialing, and touch pad based remote controller, are developed to make the proposed videophone system more intuitive and easy to use for the elderly. Based on Support Vector Machine (SVM) algorithms, an evaluation model is also developed with the data collected from the embedded multimedia device. It is useful to evaluate the physical and psychological health of the elderly.