by Glen Emerson Morris
Considering Colorado has been the home of three of the most publicized computer disasters in the US (the airport baggage handling system, the DMV system and the Colorado Benefits Management System), it won’t be surprising if these events become case studies in some future book.
If any good can come from the last two computer disasters it would be that important lessons are learned about quality assurance. Both incidents were avoidable, especially the Colorado Benefits Management System fiasco. Advertisers can avoid having similar problems with their Websites if they use a little common sense and they rely on experienced computer professionals.
In the case of the Colorado Benefits Management System, the biggest mistake made was that there was no provision for a rollback to the older, and working, system if the new system failed. Even for minor changes on existing Websites, provision for rollbacks should always be made. In the case of setting up a new system, rollback provisions should be mandatory, and the rollback process should be tested before the new system goes online.
Another probable mistake was the apparent failure to do load testing which would have alerted EDS to the slowness of the system and the problems that would lead to. It is true that load testing is relatively expensive and requires specialized engineering talent, but it’s usually worth it. So is automated testing for functionality and usability. A ballpark estimate for load testing software would be about $50,000. Add another $200,000 for automated testing software and the total cost for all automated testing software would be about $250.000, not counting the hardware required to run it. However, given the $200.000.000 cost of the Benefits Management System, the cost of automation software would have only been slightly more than one tenth of one percent of the total cost.
Automated functionality testing would have made EDS aware that the Colorado Benefits Management System ran so slowly that data entry timed out when moving between data entry screens. According to some reports 17 different screens had to billed out and it took up to 24 minutes for each new screen to load. There are several possible reasons for the system running this slow, some related to hardware, and some related to software, and all of them are avoidable. Hardware issues could be from using processors that were too slow, or didn’t have enough memory installed, or were configured improperly. Software issues could be from improper cache settings, improper software configuration, or the much worse issue of a fundamentally bad software design in the first place.
Yet another problem was requiring 17 different screens to process one case. This is a design error that would have never been allowed to happen by experienced software developers. This also suggests that no usability testing was done. This aspect of the system, combined with its extreme slowness, made the system fundamentally unusable.
Extreme work hours may have been another factor. In all probability, the EDS engineers working on the Colorado Benefits Management System were putting in far more than 40 hours a week, and the resulting disaster was at least partly due to this factor.
Since the dot com crash, companies have generally required much longer hours from employees than before. Good software engineering requires many variables be considered simultaneously. However, the ability to multitask is one of the first abilities people lose when they become tired. (This is the reason the FAA limits pilots to flying only 1200 hours a year. Beyond that time, pilots are much more likely to make mistakes, and serious ones.) As management guru Peter Drucker once commented. “Never confuse motion with productivity.”
According an industry insider, EDS has a history of hiring bright new graduates, paying them as little as possible and letting them learn on the job. The result is a workforce with intelligence but little experience. Unfortunately, this is an industry wide problem.
Software quality assurance hasn’t been the same since the dot com crash and 9/11 attack. The software industry responded to the market crash by budget cuts that targeted many of the higher paid, and most senior, professionals in the industry. The result is that nearly everyone who knew what they were doing has been laid off from the software industry.
Probably the biggest example of this is the case of Robin Hedges (AKA Robin Sterling), who founded Apple’s first QA division, and was later Director of QA for Radius, Adobe and eBay. Considered by many to be the best QA manager in the business, she was unable to find employment after the crash and eventually gave up and opened a hat shop. Her case was not alone.
A former manager for a major network security products developer cited a case where a former Apple automation engineer was laid off from a team because he was higher paid than the others on the team. The problem was that the former Apple engineer could write automation code in four hours that would take anyone else on the team two weeks to write, if they could write it all, and his code always seemed to work the first time it was tried.
There has been a management problem in the software industry for decades because most engineering managers have engineering degrees, not management degrees. Unfortunately, the skills required to manage an engineering department are vastly different than those needed to program code, and this usually shows up in the quality of the product.
Every mistake made in the development and deployment of the Colorado Benefits Management System was completely avoidable. A little previous experience and common sense would have avoided a situation that adversely affected tens of thousands of people. The real tragedy is that, given the state of the American software industry, the mistakes made weren’t surprising at all.
Glen Emerson Morris has worked as a technology consultant for Network Associates, Yahoo!, Ariba, WebMD, Inktomi, Adobe, Apple and Radius, and is the developer of the Advertising & Marketing Review Data CD.
