The challenges of cyber security have outpaced the advantages of cyber tools and technologies. In 2018, World Economic Forum has already placed cyber security in the top five risks faced by the world. Cyber threats are evolving and can cripple economies and nations. The major tools of cyber threats are anonymity, deception and uncertainty. Current state of the art research is also evolving into addressing these challenges by applying new and proactive threat hunting approaches instead of doing reactive cyber defense, which is proving futile. Malware is an indispensable tool of cyber threat actors to accomplish malicious activities i.e. exfiltration, espionage and disruption. Using advanced obfuscation and mutation methods, malware adversaries are able to remain ahead of cyber defenders. Most malware detection technologies are based on finding a-priori known signatures of malware payload or known patterns of malware behavior. This dissertation addresses the challenge of hunting unknown behaviorally mutated malware inside a host computer by proposing a proof of concept framework named Malvidence for characterizing malware behavior within a host operating system process tree using cognitive machine intelligence. Using Malvidence framework, tools and techniques can be derived for variety of cyber security methods for threat detection. Cognitive Computing is a promising domain of machine intelligence which explores and develops new tools to incorporate human cognitive characteristics so that the performance of existing domain of artificial intelligence and machine learning can be improved. Therefore, cognitive complexity based fractal analysis is demonstrated and a methodology of extracting inherent but hidden patterns of malware dynamics using a temporal graph theoretical approach is proposed. Further, a set of graph theoretical features is analyzed and proposed for an effective characterization of malware behavior which can be subsequently used for malware hunting and detection. In addition, the proposed features are tested for their mathematical validity. Finally, using proposed cognitive complexity analysis, characterization performance of an unsupervised clustering algorithm is provided to demonstrate the validity of Malvidence framework.