COCOMO and the Courtroom: A Brief Case Study
Having a strong knowledge of the COCOMO II estimating model is beneficial for software planning and estimation, but it can also play a key role in the outcome of a legal dispute regarding the estimate to complete a large software system already in development. Recently, WSR Consulting Group, LLC, used COCOMO II with the project realities uncovered in discovery to determine how much time, effort and cost the Systems Integrator would need to deliver this large ERP system.
In 2007, a State signed with a Systems Integrator (“Integrator”) to create a federally mandated web-based management system for their Family Court and Children’s Support Services departments. The original contracted-for schedule was a 36-month implementation at a cost of $50 million. What followed was a series of mis-steps, miscommunications, mis-management, and mistakes.
Instead of meeting the promised milestone Go-Live date of Spring 2010, the Integrator extended the schedule again and again, pushing it from 36 months to 83 months. When even several months of delayed Systems Testing, a critical milestone deliverable, had only been 30% completed less than a month from the latest extension deadline, the State had no choice but to terminate the contract and pursue a lawsuit in an attempt to recoup the monies they had spent. WSR Consulting Group (“WSRcg”) was brought in to opine for the court about the quality of the work and to estimate the time the Integrator would need to complete the project.
WSRcg followed its proprietary protocol to develop a thorough understanding necessary to provide a clear, straightforward, and persuasive expert report and opinion. During document discovery reviews, customer interviews, industry research, deposition reviews and a thorough examination of the system’s project management, planning, execution, status and quality, our experts found the following:
1. Poor project management: Steering committee was untrained for their role; status reports were “rosier than actual”; stakeholder expectations were not managed; 75% of promised “super managers” never showed up; misleading project metrics.
2. Immature software development lifecycle (SDLC) processes: a lower level CMMi 1; promised SDLC best practices were abandoned arbitrarily to meet deadlines; system quality and risks poorly managed.
3. Staff capability: inexperienced in the industry, technology, integrator’s standards; untrained in tools; high turnover of staff.
4. Poor use of systems development and testing tools: no formal or automated regression testing performed; no updated requirements traceability matrix (RTM); much work was subbed to India with limited controls or oversight.
5. Bad Integrator/Customer Relationship: Customer distrustful of progress reports and project status; Integrator continually turned over incomplete deliverables to customer for the latter to find and correct problems.
6. System needs: Highly reliable; with accurate data; easy-to-use web-based system for users; following hundreds of pages of regulations.
Here are the opinions presented in testimony and issued as part of our expert report:
Opinion 1: The Systems Integrator did not meet amended contract schedules before and during the key Systems Test completion deadline, even though it shortcut many of the required tasks. At the time of State termination, completion date lay “well beyond” the five times extended statewide-Go-Live date.
Opinion 2: Because of Integrator’s failings, the Systems Test proceeded at a very slow pace revealing large numbers of basic functional gaps, design defects, coding mistakes, and a lack of internal system quality and controls.
Opinion 3: It is impossible to determine the project’s true status at termination, nor when the Integrator would complete it. Low quality software meant end-to-end business process testing (Systems Functional Test & User Acceptance Test) would be mired in significant defects and delays.
Opinion 4: At Termination, the Systems Integrator could not submit a reliable Corrective Action Plan because of failings in requirements elicitation and analysis, project management, SDLC deviations, and inadequate and incomplete testing. The Integrator would have to first acknowledge the true causes of the project’s low quality software and delay and commit to
significant advances in project performance, staff productivity, and timely, high-quality deliverables.
Opinion 5: The Integrator has not demonstrated ability or willingness to finish this project in a reasonable amount of time. Its ad hoc, reactive project planning/execution will not work to develop a suitable, maintainable system in the foreseeable future. State had ample grounds to lose confidence in Systems Integrator’s ability (did they have the promised and needed expertise, processes, tools and staff?) and willingness (would it assign those assets to this project?) to make improvements to complete project with sufficient quality in reasonable time.
Opinion 6: Only a fraction of the Integrator’s deliverables (calculated by WSRcg at 27%) may be relied upon going forward with a team able/willing to perform in workmanlike manner.
At the time WSRcg was retained, the Integrator was six years into the 36 month project. The delay was, the Integrator claimed, because of the State’s interference, alleged scope increases, delays in approvals, their customer changing its mind after approvals, understaffing its promised business analysts and changing requirements. If the State would “work with them”, the Systems Integrator told the judge they could wrap up the project in 18 months. The Judge asked, “Mr. Reid, what do you think, and why?”
The COCOMO II estimating model contains scale and cost drivers which can be altered to reflect the realities of many different factors in building a large-scale system and the necessary features of the system itself. WSRcg was able to apply the lessons learned during discovery to the COCOMO II estimating model and determine that if the Integrator continued “as-is”, (because it had demonstrated an unwillingness or inability to alter the way it was working), finishing the system would take another 10 years, and the finished product would be of low quality – negatively impacting maintenance, system reliability, and total cost of ownership.
The ten-year estimate to complete was effectively a death knell to the current project team, since the effectiveness of a product created with poorly elicited requirements, an inexperienced team, and ineffective communication would be obsolete by the time it was completed, if it functioned beyond a marginal level at all. Our recommendation was to terminate the contract, use the less than 27% of salvageable artifacts, and re-start with a more experienced, expert, capable, industry-savvy team building on an available and already-proven core CSES system.
The judge accepted the WSRcg estimate and testimony as reliable evidence based on industry leading best practices as performed by an expert in COCOMO II, project management, and SDLC, using data and variables appropriately collected and analyzed during project investigation and discovery. The State was awarded a financial judgment and is now working with another Integrator to complete the project.
By WSR Consulting Group, LLCABOUT THE AUTHOR: Warren S. Reid
Computer Projects and Software Failure Consultants & Expert Witnesses
Computer Projects and Software Failure Consultants & Expert Witnesses
Warren S. Reid possesses 40+ years of progressive experience as a management-, litigation- & technology-consultant, and expert witness. As Managing Director of WSR Consulting Group, LLC since 1988, he is sought after worldwide for his expertise in: discovering the root causes and parties responsible for software project failure; evaluating the execution of promised SDLC methods; analyzing assigned project team skills vs. contract responsibilities; assessing project estimate correctness/management; managing risk; judging PM leaders & practices; evaluating the test regimen, plan vs. actual, test results patterns and QA; predicting Go-Live chaos based on short cuts taken, and more. He has been an expert witness in IT disputes in North America, Asia and Europe testifying in U.S. State & Federal Courts, U.S. Court of Federal Claims, the International Arbitration Center in London, and more. He has a B.B.A., M.S. & an M.B.A. in Management Information & Control (from Wharton Graduate School).
Copyright WSR Consulting Group, LLC
Disclaimer: While every effort has been made to ensure the accuracy of this publication, it is not intended to provide legal advice as individual situations will differ and should be discussed with an expert and/or lawyer.For specific technical or legal advice on the information provided and related topics, please contact the author.