I E E E S o f t w a r E P u b l i s h e d b y t h e I E E E C o m p u t e r S o c i e t y0 7 4 0 -7 4 5 9 / 1 0 / $ 2 6 . 0 0 © 2 0 1 0 I E E E focus management hardly help increase project success. Over the years, their figures have attracted tremendous attention.However, we question the validity of their figures. Robert Glass 2,3 and Magne Jørgensen and his colleagues 4 indicated that the only way to assess the Chaos results' credibility is to use Standish's data and reiterate their analyses. But there's another way: obtain your own data and reproduce Standish's research to assess its validity. We applied the Standish definitions to our extensive data consisting of 5,457 forecasts of 1,211 real-world projects totaling hundreds of millions of euros. Our research shows that the Standish definitions of successful and challenged projects have four major problems: they're misleading, one-sided, pervert the estimation practice, and result in meaningless figures.
Misleading DefinitionsThe Standish Group published the first Chaos report in 1994, which summarized Standish's research findings and aimed to investigate causes of software project failure and find key ways to reduce such failures. 1 The group also intended to identify the scope of software project failures by defining three project categories that we recall verbatim:■ Resolution Type 1, or project success. The project is completed on time and on budget, offering all features and functions as initially specified. ■ Resolution Type 2, or project challenged. The project is completed and operational but over budget and over the time estimate, and offers fewer features and functions than originally specified. ■ Resolution Type 3, or project impaired. The project is cancelled at some point during the development cycle. 1To find answers to their research questions, Standish sent out questionnaires. Their total sample size was 365 respondents representing 8,380 applications. On the basis of the responses, Standish published overall percentages for each project cat-F or many years, researchers and practitioners have analyzed how to successfully manage IT projects. Among them is the Standish Group, which regularly publishes its findings in its Chaos reports. In 1994, Standish reported a shocking 16 percent project success rate, another 53 percent of the projects were challenged, and 31 percent failed outright.