Blog

I’ve been curious for a while how Microsoft’s latest database offering stacks up against the competition. So I did some searching today, and unfortunately couldn’t find any good results. The problem is that most DB people don’t understand how to do good benchmarking. Most people point to the Transaction Processing Council’s database of benchmarks, but these are not valid benchmarks. Why? Because they don’t standardize the hardware. Their goal is to show how fast you can run a database, without taking into account cost and personal bias. The results in their database are often from servers that cost millions of dollars. This would be fine if every test was from the same million dollar hardware, but it’s not. So there are way too many competing factors.Good benchmarks test competing systems on the same exact hardware, in the same exact lab environment, with the same exact tests for all systems. That way the only variable is the system you are testing. TCP’s tests have a large array of variables, so there’s no way to know why one system performs better than the other. So I’m going to have to keep my eyes peeled for a true benchmark. I did see one that compared SQL Server 2000 to Oracle, DB2, and MySQL, and it was a true benchmark (MSSQL paled in comparison). But I have yet to see one for MSSQL 2005.