No abstract

To protect core functions, applications often utilize the countermeasure techniques such as antidebugging to avoid analysis by outsiders, especially the malware. Dynamic binary instrumentation is commonly used in the analysis of binary programs. However, it can be easily detected and has stability and applicability problems as it involves program rewriting and just-in-time compilation. This paper proposes a new lightweight analysis method for binary programs with the assistance of hardware features and the operating system kernel, named BAHK, which can automatically analyze the target program by stealth and has wide applicability. With the support of underlying infrastructures, this paper designs several optimization strategies and specific analysis approaches at instruction level to reduce the impact of fine-grained analysis on the performance of target program so that it can be well applied in practice. The experimental results show that the proposed method has good stealthiness, low memory consumption, and positive user experience. In some cases, it shows better analysis performance than the traditional dynamic binary instrumentation method. Finally, the real case studies further show its feasibility and effectiveness.

At present, various vulnerabilities and malicious programs are still constantly threatening the system security, and in-depth analysis of legitimate applications and malicious code is an important link of security defense under the current security situation. Dynamic binary analysis is the important method in the field of binary analysis, and after years of research and development, many valuable results have been achieved, and some mature tools generated have been widely used in practice. But it still faces multiple challenges in terms of analysis efficiency, deployment, and the impact on the target program. Based on the existing research, this paper proposes a new lightweight hypervisor-based dynamic binary instrumentation method, which uses the virtualization features of new processors to perform transparent and efficient execution interception of the target program, so that it can instrument the target program and redirect the original execution. To take full advantage of the interception solution, we further present a compact inline dynamic taint analysis method that enables fine-grained data flow analysis of the target program with multiple processes and kernel modules. Experiments in the real environment show that the proposed method is effective and efficient in the binary instrumentation and analysis, it can achieve a good balance in analysis performance, availability and stealthiness, and can be applied to the practical analysis scenarios.

Many famous attacks take web browsers as transmission channels to make the target computer infected by malwares, such as watering hole and domain name hijacking. In order to protect the data transmission, the SSL/TLS protocol has been widely used to defeat various hijacking attacks. However, the existence of such encryption protection makes the security software and devices confront with the difficulty of analyzing the encrypted malicious traffic at endpoints. In order to better solve this kind of situation, this paper proposes a new efficient and transparent method for large-scale automated TLS traffic analysis, named as hyper TLS traffic analysis (HTTA). It extracts multiple types of valuable data from the target system in the hyper mode and then correlates them to decrypt the network packets in real time, so that overall data correlation analysis can be performed on the target. Additionally, we propose an aided reverse engineering method to support the analysis, which can rapidly identify the target data in different versions of the program. The proposed method can be applied to the endpoints and cloud platforms; there are no trust risk of certificates and no influence on the target programs. Finally, the real experimental results show that the method is feasible and effective for the analysis, which leads to the lower runtime overhead compared with other methods. It covers all the popular browser programs with good adaptability and can be applied to the large-scale analysis.