Dueling SPC Benchmarks

by Administrator on January 30, 2008

I love it when vendors decide to fight out their performance claims at the Storage Performance Council.  Thanks to Pq65, I have been pointed to a flock of Network Appliance benchmarks recently posted to SPC in which the vendor both presented the performance of its own box, the FAS 3040 with and without Snapshot enabled, and their competitor’s box, the EMC CX3 Model 40, with and without SnapView enabled, which NetApp apparently bought, configured and tested of its own volition and on its own dime.

EMC’s gear got clobbered, of course.

Now, here’s a question:  doesn’t NetApp have to disconnect the CX3 and use it as a door stop for violating the warranty agreement on the EMC product that states that the warranty is nullified if you talk publicly about the performance of the gear without EMC’s okay?  If the tables were turned, NetApp’s similar codicil would not only remove the warranty but also invalidate the software licenses on the box, prohibiting the user from connecting the FAS to a TCP/IP network.

Hmm.

I invite everyone to read the comparative reports and tell me what they think about dueling SPC benchmarking as a marketing ploy.

FAS3040.pdf

FAS3040 with Snapshot.pdf

CX3 Model 40.pdf

CX3 Model 40 with SnapView.pdf

Let the revels begin.

{ 4 comments… read them below or add one }

Pq65 January 30, 2008 at 5:20 pm

Nope. Apparently Chuck extended an open invitation on his blog a while back to everybody who was able and willing to run the SPC1 on the clarion.

“We’ve never done an SPC test, and probably will never do one. Anyone is free, however, to download the SPC code, lash it up to their CLARiiON, and have at it.”

So netapp gets a warranty pass. :-)

ValB January 31, 2008 at 1:05 am

Actually, EMC already published their un-audited and highly fictional findings of NetApp performance back in late 2006. We think the desperation of the effort speaks for itself so we didn’t bother to respond legally.

I covered it on my blog here: http://blogs.netapp.com/exposed/2007/12/benchmarking–1.html

lrazo January 31, 2008 at 4:25 am

Hi Jon,

I’ve been a NetApp employee for a little over 10 years, and I can assure you that EMC has been publishing and distributing unverified and unvalidated NetApp benchmarks for years. You can still get one of the more recent examples here:
http://www.mediafire.com/?50ddff9a5rc

Unlike the SPC-1, this report and others like it are created in the privacy and sanctity of EMC’s own labs, are never verified by any third party and generally leave all kinds of key details out of the report.

To my knowledge, however, our legal team has never gone after EMC – or for that matter, any of the other many competitors like HP, Dell and others who have regurgitated the same FUD and placed their logo on it.

I’ve been a long time reader and fan of your blog and I know that this point about the NetApp EULA is a big pet peeve, however I think its purpose has been generally misunderstood. In my 10 years at NetApp (in a wide variety of customer-facing roles) I have never heard of even one case where anyone was pursued over that clause.

Now taking my NetApp hat off for a moment and speaking only for myself — I’ve read the clause and it does not restrict any customer from sharing their experiences with other customers or prospects or anyone else. The clause restricts creating publicity. To me it reads as though it is meant to allow NetApp to reserve the right to protect itself from just the kinds of misleading (and sometimes even bordering on slanderous) publications from our competitors.

That being said, I think it speaks volumes about our company that in spite of all those options, we’ve _still_ chosen to take the high road and clear things up instead by publishing an independent, audited, and widely respected benchmark like the SPC-1. I think the fact that these results are also completely in line with all of our previous benchmarks and performance reports (such as Avanade Exchange Report, SPEC SFS, ESRP, VeriTest, NetApp TR 3521, and others) also validates our claim in general.

There is no doubt, NetApp has become a large company, but being one of the “old timers” from the days of NetApp as a start-up, I think we are still a company that takes its values seriously and will always try to do the right thing. This is exactly why I have stayed around for so long and will probably be around a good while more.

InsaneGeek January 31, 2008 at 1:13 pm

I don’t believe any storage vendors claims of performance period, SPC-1, SPC-2, SPECFS, iops/sec, MB/sec, etc. are all useless and provide no benefit at all to me as a customer (maybe some bathroom reading, I do enjoy the laughable claims between vendors sometimes). Only putting it on my floor and actually testing it gives me anything to believe period.

I do have a question as to whether or not the way they ran the SPC-1 benchmark (or if it does it in general) if it read or write to the data/user/log ASU storage at the same time. They made 3x luns per raid group (one for each ASU), and then striped across all raid groups, so a read/write to any one would require a fairly substantial head move between ASU groups (which would not be how any sane person would be configuring the array).

Previous post:

Next post: