Pilot Implementation: Learning from Field Tests in IS Development

Abstract

A recurrent problem in information-systems development (ISD) is that many design shortcomings are not detected during development, but first after the system has been delivered and implemented in its intended environment. Pilot implementations appear to promise a way to extend prototyping from the laboratory to the field, thereby allowing users to experience a system design under realistic conditions and developers to get feedback from realistic use while the design is still malleable. We characterize pilot implementation, contrast it with prototyping, propose a fiveelement model of pilot implementation and provide three empirical illustrations of our model. We conclude that pilot implementation has much merit as an ISD technique when system performance is contingent on context. But we also warn developers that, despite their seductive conceptual simplicity, pilot implementations can be difficult to plan and conduct. It is sometimes assumed that pilot implementations are less complicated and risky than ordinary implementations. Pilot implementations are, however, neither prototyping nor small-scale versions of full-scale implementations; they are fundamentally different and have their own challenges, which will be enumerated and discussed in this article.

Grappling with the future: The messiness of pilot implementation in information systems design

“…A method for making design ideas manifest and for simulating use in the laboratory (i.e., away from real work) in order to traverse a design space and test the fit between a proposed design and the user 8 Pilot implementation A method for learning about the fit between a system and its context by employing a properly engineered, yet unfinished, version of the system for real use in its intended environment for a limited period of time 9 Continuing design An approach for gradually evolving a system after it has entered fullscale use by providing on-site technical support to assist users in tailoring the system to their evolving needs and in maintaining a working configuration of systems 10…”

“…An approach for evolving a system after it has entered full-scale use by periodically releasing new versions developed to exploit new technological possibilities and incorporate feedback on the use of previous versions 11 Hertzum et al 9 define pilot implementation as "a field test of a properly engineered, yet unfinished system, in its intended environment, using real data, and aiming-through real-use experience-to explore the value of the system, improve or assess its design, and reduce implementation risk" (p. 314). This definition makes pilot implementation a meeting ground between the system and its environment and, thereby, an opportunity for learning about the mutual adaptations needed to make the system a success.…”

“…[12][13][14] Pilot implementations supply such practical experience, but their limited scope and the unfinishedness of the pilot system necessitate preparations to enable real use and safeguard against errors. 9 We will analyze pilot implementations from the point of view that they are situated and messy activities. They are situated because the particulars of the pilot site influence the pilot implementation, irrespective of whether these particulars are representative of the conditions under which the finished system will subsequently be used.…”

“…The defining characteristic of pilot implementation is that the system is sufficiently functional and robust to enable testing in its intended environment but is not yet finalized. 9,15 That is, the results of a pilot implementation can affect the finalization of the technology as well as feed into its organizational implementation. A pilot implementation consists of five activities: 9  Planning and design, that is defining the pilot implementation.…”

“…9,15 That is, the results of a pilot implementation can affect the finalization of the technology as well as feed into its organizational implementation. A pilot implementation consists of five activities: 9  Planning and design, that is defining the pilot implementation. This activity involves determining where and when the pilot implementation will take place, what facilities the pilot system will include, and how lessons learned during the pilot implementation will be collected.…”