System Call Monitoring Using Authenticated System Calls

Abstract

System call monitoring is a technique for detecting and controlling compromised applications by checking at runtime that each system call conforms to a policy that specifies the program's normal behavior. Here, we introduce a new approach to implementing system call monitoring based on authenticated system calls. An authenticated system call is a system call augmented with extra arguments that specify the policy for that call, and a cryptographic message authentication code that guarantees the integrity of the policy and the system call arguments. This extra information is used by the kernel to verify the system call. The version of the application in which regular system calls have been replaced by authenticated calls is generated automatically by an installer program that reads the application binary, uses static analysis to generate policies, and then rewrites the binary with the authenticated calls. This paper presents the approach, describes a prototype implementation based on Linux and the PLTO binary rewriting system, and gives experimental results suggesting that the approach is effective in protecting against compromised applications at modest cost.

Securing untrusted code via compiler-agnostic binary rewriting

“…By positioning the rewritten versions of these two functions at those addresses, the rewriter can encode overlapping pointers to them in the lookup table. With chunk size c = 16 and memory division d = 2 28 , a rewritten code base address of 2 24 (t & 0xF) + 2 16 t supports at least 15 two-pointer collisions and 1 three-pointer collision per rewritten code page-far more than we saw in any binary we studied.…”

“…Past work [17,19,28] has shown that IRM systems are capable of enforcing more sophisticated temporal properties when equipped with more powerful event languages and responses to impending policy violations that go beyond mere program termination. Developing policy-enforcement libraries that implement such policies is therefore a logical next step toward applying our framework to interesting, practical security problems for these real-world systems.…”

Using static analysis for Ajax intrusion detection

“…In addition, they apply binary rewriting to assign unique names to call sites, to further curb nondeterminism in the monitor (an analogous technique, call-signing, is presented by Rajagopalan et. al [26]). Because invocations of xhr are, however, often deep within frameworks, signatures cannot be placed at invocations of that function; they must instead be "pushed up" the call chain to where the requests are really made.…”

“…Other systems use a model of calls constructed through static analysis [33,7,12,26]. These systems all address operating system monitoring, where the operating system (corresponding to our server), monitor, and application (corresponding to our browser client) all run on the same machine.…”

Identifying native applications with high assurance

“…A more usable MAC solution is described in [16]. A cryptographic-based MAC system is the authenticated system call work by Rajagopalan et al [22], which is closed in spirit to our A2 framework. The presented work in the authenticated system call is limited to providing identities (the HMAC) to individual function calls to system calls in an application.…”