Vital signs monitoring systems are rapidly becoming the core of today's healthcare deliveries. The paradigm has shifted from traditional and manual recording to computer based electronic records and further to handheld devices as versatile and innovative healthcare monitoring systems. Interpretation of vital signs to early detect multiple physical signs using a multifactorial and holistic approach is presented. A total of 30 patient data have been collected under local and national ethics approvals in New Zealand Hospitals. Ultimately, this system achieved a high level of agreement with clinicians' interpretation when assessing specific physical signs such as bradycardia, tachycardia, hypertension, hypotension, hypoxaemia, fever and hypothermia. The proposed vital signs interpretation system was validated through off-line as well as real-time tests with a high level of agreement between the system and the physician. The system achieved an accuracy of 95.83%, sensitivity of 100%, specificity of 93.15%, and predictability of 90.38% in compare with a clinician assessment for tachycardia, hypertension, hypotension, hypoxaemia and hypothermia.