Heart sounds and heart rate (pulse) are the most common physiological signals used in the diagnosis of cardiovascular diseases. Measuring these signals using a device and analyzing their interrelationships simultaneously can improve the accuracy of existing methods and propose new approaches for the diagnosis of cardiovascular diseases. In this study, we have presented a novel smart stethoscope based on multimodal physiological signal measurement technology for personal cardiovascular health monitoring. The proposed device is designed in the shape of a compact personal computer mouse for easy grasping and attachment to the surface of the chest using only one hand. A digital microphone and photoplehysmogram sensor are installed on the bottom and top surfaces of the device, respectively, to measure heart sound and pulse from the user’s chest and finger simultaneously. In addition, a high-performance Bluetooth Low Energy System-on-Chip ARM microprocessor is used for pre-processing of measured data and communication with the smartphone. The prototype is assembled on a manufactured printed circuit board and 3D-printed shell to conduct an in vivo experiment to test the performance of physiological signal measurement and usability by observing users’ muscle fatigue variation.