ReScan: A Middleware Framework for Realistic and Robust Black-box Web Application Scanning

Abstract

Black-box web vulnerability scanners are invaluable for security researchers and practitioners. Despite recent approaches tackling some of the inherent limitations of scanners, many have not sufficiently evolved alongside web browsers and applications, and often lack the capabilities for handling the inherent challenges of navigating and interacting with modern web applications. Instead of building an alternative scanner that could naturally only incorporate a limited set of the wide range of vulnerability-finding capabilities offered by the multitude of existing scanners, in this paper we propose an entirely different strategy. We present ReScan, a scanner-agnostic middleware framework that transparently enhances scanners' capabilities by mediating their interaction with web applications in a realistic and robust manner, using an orchestrated, fully-fledged modern browser. In essence, our framework can be used in conjunction with any vulnerability scanner, thus allowing users to benefit from the capabilities of existing and future scanners. Our extensible and modular framework includes a collection of enhancement techniques that address limitations and obstacles commonly faced by state-ofthe-art scanners. Our experimental evaluation demonstrates that despite the considerable (and expected) overhead introduced by a fully-fledged browser, our framework significantly improves the code coverage achieved by popular scanners (168% on average), resulting in a 66% and 161% increase in the number of reflected and stored XSS vulnerabilities detected, respectively.

APIMiner: Identifying Web Application APIs Based on Web Page States Similarity Analysis

“…The experiment described below was conducted to verify the validity of our similarity analysis method and identify the ideal weight distribution and threshold for our web page similarity analysis method. Inspired by the study of Kostas et al [17], we built the test data set using posts in WordPress. In total, we prepared the following three data sets, two of which consisted of pages we manually compiled:…”

“…Consequently, this paper abstained from conducting an experimental comparison of this method type. We compared the similarity analysis method proposed in this paper with state-of-the-art similarity analysis methods, including NDD (Normalized DOM-edit Distance) and mNDD [17].…”

“…It then establishes the difference between two trees by computing the edit distance. The mNDD method, a variant of NDD, was proposed by Kostas et al [17]. In the process of constructing corresponding trees from the DOMs of pages, mNDD recursively discards leaf nodes devoid of functionality (e.g., tags for line breaks, paragraphs, spans, divs, or font formatting) while preserving nodes associated with functionality (e.g., scripts, forms, iframes, buttons, and inputs).…”

“…In this context, a smaller edit distance between two page states implies a higher degree of similarity between them. We obtained the NDD and mNDD algorithms from the open-source projects of Vissers et al [16] and Kostas et al [17] for the experimental evaluation.…”

“…We conduct extensive experiments to verify the effectiveness of APIMiner by comparing it with state-of-the-art methods such as NDD [16] and mNDD [17]. The experimental results demonstrate that our method surpasses existing state-of-the-art methods in accurately and finely distinguishing between similar and dissimilar pages.…”