Last year, I worked as part of a project management office for one of the biggest defense contractors in the world. I was a contractor myself, getting paid by the hour to help them with project planning, forecasting, status, and other PMO and IT advisory functions. So when IBM conned them out of millions of dollars, I was sitting right in the front row.
Thanks to Gulf War II, this defense contractor was growing like mad, despite the fact that it already had 120,000 employees and $30 Billion in Revenue. It now had the financial resources to take on major internal IT projects that it had been dreaming about for years. I worked on one of these, an enterprise-wide Portal and Knowledge Management project.
One of the criticisms of Knowledge Management is that it is just a new way to sell the same old products. Vendors take an online message board, bundle it with an off-the-shelf document management system and an intranet search, call the whole thing a "knowledge management system", and then sell it to you for millions. My take on knowledge management is that it's a good idea that's been overhyped and oversold, like Business Intelligence or Customer Relationship Management.
When I started on the project, they had already selected most of the vendors. Opentext for document management. Verity for Search. Microsoft for Directory Management. Netegrity for single sign-on. And IBM for the Portal.
There was one IBM consultant on site, and a second would show up occasionally. The first was a project analyst. His stated job was to make sure that all of the software interoperated with the portal, and to help with the overall project planning. But we never got any real planning or analysis out of him. He spent most of his time selling.
Vendor selection, when done properly, is very difficult and time-consuming. This IBM consultant's job was to stop the process in its tracks. Whenever management was trying to select a vendor, or even having second thoughts about a vendor, this consultant would offer senior management the solution to their problems. Management, in a hurry, would agree, happy to have the matter resolved. The questions of whether IBM could deliver on their promises or whether their bid was competitive went unasked.
The second consultant's job was more sinister. He was a thought leader.
He showed us how other companies had applied these same tools and saved millions of dollars. These were some of the highest quality presentations I'd ever seen - as good as anything I saw in my MBA program. They were compelling and intellectually cohesive. And just before he left, he invited us to a recorded teleconference by his guru, Larry Prusak. Prusak is a thought leader for IBM who has published several books on knowledge management, and now runs their knowledge management practice.
Before long, IBM components were replacing many of the other vendors' components in our planned architecture. Somehow, we even ended up with an installation of DB2. The problem was that we had no idea what they would cost us to implement, or whether they would work at all.
Where was the technical staff during all of this? Staying out the way, mostly. They knew that IBM was selling solutions two levels over their head, and they didn't want any part of it. They certainly didn't want implementation responsibility - they didn't want to be on the hook for delivering whatever IBM was promising. And IBM was more than happy to leave them out the process.
Of course, it was my job to tell management how much this Portal/KM project was going to cost, which meant getting estimates from IBM and the technical team. IBM lowballed their estimates, but never provided specific numbers and plenty of assumptions - i.e. "If everything goes well, most of these tasks shouldn't take more than a few weeks". Our internal technical team was reluctant to make any estimates at all, because they certainly didn't want implementation responsibility. Eventually, when they were forced to provide estimates, they were sky-high and also loaded down with assumptions.
As the project planning dragged on, our self-imposed deadline became closer and closer. With an expanding amount of work and a shrinking amount of time, that meant that the amount of resources needed was growing every day. Upper management was very reluctant to move back the deadline because the project had a lot of visibility, and executive bonuses were dependent upon completing the project by the end of the year. The decision was made to move forward on anything we could by using IBM resources, even though project planning was not fully completed (because of the lack of estimates and lack of input from the original consultant).
At this point, IBM Global Services consultants flooded our conference rooms. Overnight, we ended up with twenty consultants. When I asked how much these consultants were costing us, I was told $250/hr. This information proved to be incorrect - they were actually charging us $325/hr.
What were we getting for $325/hr? People hired off of Monster and Careerbuilder. Seriously.
Management was under the assumption that we would be getting real implementation experts from IBM. In fact, we were getting employees from a subcontractor. We paid IBM $325/hr, and they paid their subcontractor about $165/hr. The subcontractor then paid its people salaries of $90,000 to $110,000/yr, the market average, which equates to about $75/hr when benefits are included. We were paying a markup of about 333%.
Were these people experts? A few were. Most were just Java programmers or Websphere administrators. And a few were essentially useless. It was a fairly typical distribution of employees - some stars, most fairly average, a little dead weight.
I know the contract market fairly well, given that I used to be a contract Java programmer myself, and that I still have a few friends who still do this kind of work. The average technical contractor makes between $50/hr to $100/hr, depending upon his or her experience, certifications, skill set, and the going rate in the region. Most of these people work for contract agencies such as Modis and RHI that add another $30/hr to the bill rate. If you're good enough to find your own clients and become an independent contractor, you can keep that markup for yourself and make over $100/hr.
So when I saw these rates, I was shocked because I knew a half dozen people who could have done a better job for $80/hr to $100/hr. But senior management was too nervous to use independents to complete the project - they saw the approach as a higher risk, even though they could have gotten four times as much consulting work for the same price. IBM had sold them on the assumption that experts could complete the project faster, and then had pulled the "bait and switch", by providing commodity technical staff rather than anyone truly exceptional.
One other thing that I discovered is that owning an IBM subcontractor is a nice way to make a living. The person who owned the subcontracting company we worked with was a former IBM salesperson. He calculated that he could make a lot more money by just hiring dozens of technical staff off of Monster and Careerbuilder and then selling them to IBM for $165/hr, and that's what he does now. We estimated that he was making $90,000 a week on our contract for essentially just hiring people and doing payroll.
Our budget was toast. Burnt blackened toast. We were so far over budget that we felt sick just talking about it.
We had expected IBM to stay for about three months, which all by itself would have blown our budget, given their $325/hr bill rate. But they were in our company for more than seven months, burning through more than a quarter million dollars a week. And Global Services wasn't the entirety of the IBM damage. We still had licensing and support fees for Websphere, Websphere Portal, Websphere Content Management, Tivoli Access Manager, and DB2.
IBM, which had promoted itself to lead vendor and integrator, had overpromised, overcharged, and underdelivered. We ended up with an overly complex enterprise portal with a few off-the-shelf portlets and a few integrated applications. Many application integration efforts had to be abandoned. It's unlikely that those apps will ever be in the portal, and the jury is still out on whether the portal will be a success. None of those slick knowledge management presentations we saw at the beginning of the project bore any resemblance to our outcome, and that original consultant was nowhere to be found.
What Went Wrong
There's no question that our senior management made major mistakes in vendor selection and management. I still wonder if I could have made a better case to the executives. This was my second experience with IBM, and I knew how they operated. I raised as many warnings as I could, but ultimately because IBM was the vendor with the strongest capabilities, at least on paper, they were seen by the execs as the lowest risk choice. This led IBM to be chosen even when their product was unproven or even demonstrably inferior.
IBM sells itself as a provider of business solutions. That puts them in a position to make architecture and product recommendations. It is no surprise whose hardware, software, and services they typically recommend. After all, IBM invented FUD - Fear Uncertainty and Doubt - to deal with their competitors in the 1970s.
Even though IBM presents itself as a company with very advanced capabilities (i.e. chess-playing supercomputers), most of their customers are looking for the basics: web and database hardware and software, and competent technical staff to set it all up and keep it working. All of this is now a commodity, and companies should be paying commodity prices, not IBMs 300% markups.
Learn From Our Mistakes
They say that exceptionally intelligent people are easier to con, because they don't believe they can be conned. So if you're too smart for the following suggestions, you may need them the most. Our IT execs definitely did.
Don't take shortcuts with vendor selection or project planning. Make your vendors compete with each other during the selection process.
Never, ever, ever ask an implementation company for strategy, architecture, or product advice. They have no incentive to help you and plenty of incentives to sell you products and services that you don't need at inflated prices.
Open standards means more flexibility in vendor selection. Take advantage of this.
Know the market. Be able to calculate your resellers' costs and markup. Remember that markups alone don't add any project value.
Check resumes of individual consultants. A $250+/hr consultant should be able to walk on water, and their resume should reflect that.
Maintain a list of reliable implementation partners that includes large and small vendors, small independent contractors, and capable in-house employees. Match the talent to the project and use only proven talent on new projects.
Run small pilot projects to test vendors, technologies, architectures, etc. This can be done separately or as part of an iterative development cycle.
Run A Background Check On That Vendor
Time for my bright idea. What if several Fortune 500 CIOs started sharing information on their vendors' contract performance?
Of course, Coca-Cola wouldn't want to share information with a direct competitor like PepsiCo, but perhaps they would be willing to share information with Dow Chemical, Allstate, Bank of America, Diageo, and several other non-competitive companies. Think of the project history that fifteen huge companies could accumulate and the costly purchasing mistakes that could be avoided.
Here's the problem. An IT executive isn't going to get unbiased information about IBM's strengths and weaknesses from IBM, and they probably won't get it from a magazine or web site that IBM advertises in - and IBM advertises everywhere. So if IT execs want an independent report on a vendor, they are going to have to create one by sharing resources and information.
In the meantime, we have to remember that our IT budget expenditures are the scorecard by which the marketplace measures success and failure. If we purchase from firms that overpromise, overcharge, and underdeliver, then we shouldn't be surprised if that becomes the dominant business model in the IT marketplace. On the other hand, if we take the extra effort to find and reward the vendors that offer the most value, then other vendors will try to imitate their success, and the marketplace moves in the right direction.
Legal Disclaimer: Note that the above account is based upon only my subjective personal interpretations of events and consists of opinions, my own analysis, fair industry commentary, and criticism in the public interest.
Originally posted here. Tristan Yates can be reached here.