An Empirical Study for Security of Windows DLL Files Using Automated API Fuzz Testing

Abstract

Fuzz testing is a method that inserts an unexpected data into input of a software system and finds defects of it in order to perform security testing. In this paper, We proposed a novel methodology that performed API fuzz testing automatically and evaluated it for Windows system that most of people in the world used. We implemented an automated API fuzz testing tool that our methodology applied to. Using this tool, we experimented on 1,182 DLL files and 6,117 API functions in a system fold of Windows XP SP2. We found 177 faults in them. Among faults, 10 faults are related to control flow of a program

API Fuzz Testing for Security of Libraries in Windows Systems: From Faults To Vulnerabilites

“…The debugger traces input data in programs and detects the point where data is inserted into APIs with faults. We refer to our methodology as Automated Windows API Fuzz Testing II (AWAFTII), as it succeeds the AWAFT [2] and performs API fuzz testing automatically. The steps of the AWAFTII process are as follows:…”

“…We used that system to evaluate DLL files in the system folder (c:\windows\system32) of Windows XP SP2 [2,3]. There are 1,182 DLL files in the default system folder of Windows XP SP2.…”

“…We call our methodology Automated Windows API Fuzz Testing II (AWAFTII). This method extends our previous research [2,3] for performing API fuzz testing into the AWAFTII process. The AWAFTII process consists of finding faults using API fuzz testing, analyzing those faults, and searching for input data related to parameters of APIs with faults.…”

“…We refer to this methodology as the Automated Windows API Fuzz Testing II (AWAFTII), as it is the successor to the AWAFT [2] and executes API fuzz testing for Windows systems automatically. AWAFTII functions by finding faults using API fuzz testing, analyzing the faults, and searching the input data related to parameters of the APIs with faults.…”

“…There are various entry points for inserting invalid data, such as files [14,16], Graphic User Interfaces(GUIs) [19], networks [1,12], Application Programming Interfaces(APIs) [7], and so on [10]. In our previous research [2,3], we proposed a methodology that can execute API fuzz testing automatically in the Windows OS system and find various faults for APIs in dynamic-link library(DLL) files. Among the various types of faults, there are some faults that are related to the control flow of the software system.…”