Learning response time for WebSources using query feedback and application in query optimization

Abstract

The rapid growth of the Internet and support for interoperability protocols has increased the number of Web accessible sources, WebSources. Current optimization technology for wrapper mediator architectures needs to be extended to estimate the response time (delays) to access WebSources and to use this delay in query optimization. In this paper, we present a Multi-Dimensional Table (MDT), a tool that is based on learning using query feedback from WebSources. We describe the MDT learning algorithms, and report on the MDT learning for WebSources. The MDT uses dimensions Time of day, Day, and Quantity of data, to learn response times from a particular WebSource, and to predict the expected response time (delay), and a con dence in this prediction, for some query. Experiment data was collected from several WebSources and analyzed, to determine those dimensions that were signi cant in estimating the response time for particular WebSources. Our research shows that we can improve the quality of learning by tuning the MDT features, e.g., including signi cant dimensions in the MDT, or changing the ordering of dimensions. We then demonstrate how the MDT prediction of delay may be used by a scrambling enabled optimizer. A scrambling algorithm identi es some critical points of delay, where it makes a decision to scramble (modify) a plan, to attempt to hide the expected delay by computing some other part of the plan that is una ected by the delay. We explore the space of real delay at a WebSource, versus the MDT prediction of this delay, with respect to critical points of delay in speci c plans. We identify those cases where MDT overestimation or underestimation of the real delay results in a penalty in the scrambling enabled optimizer, and those cases where there is no penalty. Using the experimental data and MDT learning, we test how good the MDT is in minimizing these penalties.

An experience report on developing an automated web services platform

“…A variety of agent-based approaches to service execution and coordination have also been proposed (Blake 2001). Learning approaches have been successfully deployed for characterizing web service response (Gruser et al 2000) and characterizing web services (Nie et al 2002). Middleware approaches build on current message oriented and distributed object models to provide a variety of system level services and address issues such as content-routing, XML compression and management.…”

Wide Area Performance Monitoring Using Aggregate Latency Profiles

“…Now the PM can choose either iLP 1 or iLP 2 to make a prediction for the client/server pair (c, s). It is also possible that there exists strong non-random associations between iLP 1 , iLP 2 and iLP 3 . In this case, the best estimate of access latency for (c, s) is possibly obtained by aggregating iLP 1 , iLP 2 and iLP 3 into an aggregate latency profile aLP , and choosing a representative profile.…”

“…iLP c,s (t, b) represents the end-to-end delay for a request from server s at time t, given as either a real number or using T O to represent a timeout. iLP c,s comes in two flavors, similar to [3]. One flavor measures time-to-first, which depends on factors such as workload at the server and size of the requested object.…”

Data Modeling and Data Management for the Biological Enterprise

“…Learning and other techniques are needed to construct cost models (Gruser et al, 2000;Nie et al, 2001). A variety of optimization approaches are needed, e.g., performance targets; alternate sources; adaptive evaluation strategies (special issue of IEEE Data Engineering 2001, edited by Hellerstein et al, 2002a.…”