This paper introduces our solution (ISIBrno-AIMT team) to the Physionet Challenge 2022. The main goal of the challenge was a classification of heart murmurs from phonocardiographic recordings into three mutually exclusive classes (i.e., present, unknown, and not present) and to determine whether the patient's overall status is Normal or Abnormal. We propose a deep learning method that classifies whether there is a heart murmur in the phonocardiographic recording and also provides heart sound segmentation. Furthermore, the expert feature classifier assesses whether the patient's status is normal or abnormal. Our approach achieved a hidden test challenge score of 0.755 in the murmur classification task and a score of 12313 in the patient outcome classification task. Our team was ranked as 9th and 12th out of 40 teams in the official ranking for murmur and outcome classification, respectively.