PerfTTCN, a TTCN language extension for performance testing

Abstract

This paper presents a new approach to test the performance of protocols, services, or application implementations (IUT) under normal and overload situations, to identify performance levels of the IUT for ranges of parameter settings and to assess the measured performance. A performance test describes precisely the configuration of the IUT, the configuration of the network, and the load characteristics (the so called performance test configuration). PerfTTCN -an extension of TTCN with notions of time, measurements, and performance -is a formalism to describe performance tests in an understandable, unambiguous and re-usable way with the benefit to make the test results comparable. MotivationDifferent approaches for guaranteeing the end user certain levels of QoS were developed, since the non-functional aspects, in particular Quality-of-Service (QoS) aspects of distributed telecommunication services (e.g. multimedia collaboration, tele-teaching, etc.) are as important as functional aspects. They include approaches for QoS negotiation between peer user and service and network provider, QoS guarantees of communication services, QoS monitoring and management including self-adapting applications.This proposal considers QoS in the area of testing. In general, testing is a method to check whether an implementation meets certain requirements that are described in a specification. QoS testing checks the end-to-end quality of a service implementation against the QoS requirements. A specific class of QoS is that of performance-oriented QoS including delays (e.g. of a response), throughputs (e.g. for bulk data), and rates (e.g. of errors). We concentrate exclusively on performance-oriented QoS, other classes of QoS are out of the scope. Subsequently, we use the term performance instead of QoS and consequently performance testing instead of QoS testing.One of the well-established methods in testing is that of conformance testing, which is used to check that an implementation meets its functional requirements. Since conformance testing is aimed at checking purely the functional behaviour of system implementations, it lacks in concepts of time and performance. Timers are the only way to require certain time periods for test events to occur. They are used to determine whether test events occur too early, too late or not at all.While traditionally the temporal ordering and type of PDUs/ASPs have been the main target of conformance testing, we attempt here to introduce performance measurements and Quality of Service (QoS) requirements in the conformance testing. Performance measurements in a 1. Guest researcher from the University of Ottawa, Canada 2 network traditionally consists in sending time stamped packets through a network and record delays and throughput. Once such data has been collected a number of statistics can be computed and displayed. However these statistics can sometime be meaningless when the actual conditions in which these measurements have been performed are unknown. Different strategies can be used to...

Real-Time Testing with TTCN-3

“…-PerfTTCN (Performance TTCN ) [5] extends TTCN with concepts for performance testing, such as: performance test scenarios for the description of test configurations, traffic models for the description of discrete and continuous streams of data, measurement points as special observation points, measurement declarations for the definition of metrics to be observed at measurement points, performance constraints to describe the performance conditions that should be met, performance verdicts for the judgement of test results. The PerfTTCN concepts are introduced mainly on a syntactical level by means of new TTCN tables.…”

BCMP Performance Test with TTCN-3 Mobile Node Emulator

“…PerfTTCN [2] proposes architecture for three types of performance testing: testing for a server, an end-to-end service and a protocol. It introduces foreground testers, which generate test load to the IUT; and background testers, which bring the IUT into a specific load state.…”

“…However, performance tests may need several parallel components, e.g. foreground and background test components [2], monitor components etc. To control the start of test components a main control utility is needed.…”

“…However, since high load may affect correct functionality of the implementation, it is necessary to track functional correctness during a performance test [2].…”

TimedTTCN-3 Based Graphical Real-Time Test Specification

“…The need for testing non-functional properties has been discussed in literature and several real-time and performance extensions for TTCN-2++ have been proposed [18,24]. Based on these experiences TIMEDTTCN-3, [3] a real-time extension for TTCN-3, has been developed and submitted for standardization.…”