- casino online slots

Once More and Quietly, Clearly

by Administrator on November 9, 2010

Just up from my sick bed and decided to check in on Twitter — something I haven’t done in a few days.  I discovered that I had been blogged by Curtis Preston over on his site and just thought I would post a few comments in response.  You can read Curtis’ insights here.

Curtis was in the audience at the Storage Decisions San Francisco talk I gave on Why Storage Costs So #$%^&*@ Much and What We Can Do About It.  He seems to have heard only a fraction of what I said, probably because I yelled a couple of times — which is bad form and probably drowned out my key points.  Here is the preso in a nutshell.

Proposition 1: Storage Costs A Lot as a Percentage of IT hardware spend

Storage costs accelerate when total cost of ownership is considered

At this point, all heads were nodding agreement.  So, I guess we recognized the problem.  In these economic times, we need to find ways to contain costs.  Right?  That leads to the next question.

What are the cost drivers, if all of the component parts are commodity?

In high school economics, we were taught to look at vendor margins for a clue.

But the vendors just aren’t increasing their margins in a down market.  So there must be another explanation for why storage arrays continue to become more expensive despite the fact that everyone is peddling the same gear.  Right?

There are many costs to build and market an array product. I focused on three.

As readers of this column know, I take exception to lots of value add software joined at the hip to proprietary controllers.  A former EMC CEO said publicly once that this was done in part to lock out the competition while locking in the customer.  True words.

Value-add software serves many purposes, though not necessarily the consumer's needs.

I proceeded to give a couple of examples of what value-add features did to cost.  Here’s one on deduplication — comparing an on-controller implementation versus a staging of deduplication as a sharable service.

Quoting former FalconStor CEO (who I admire greatly, by the way): "Dedupe is waste management system for backup."

Doing dedupe as a freestanding software service costs $2500 per TB (Sepaton), doing it in hardware increases the cost of the commodity disk drives 100x (EMC, DataDomain, Sepaton/HDS, etc.)  Big difference, I think.

Vendors respond that one drive equals 70 -- assuming that you get data reduction anywhere near that ratio.

Another example, thin provisioning.  (This was the one that Curtis took exception to.)

Oversubscribing storage to avoid having to deal with manual capacity allocation, depending on demand forecasting algorithms to provision capacity just in time.

This sounds like a huge timesaver — something we all want.  But look what it does to the cost of the hardware…

Is it really JIT provisioning or do I, like many users, have to hold capacity in reserve as a protection against a non-forecastable demand event -- a "margin call" if you will?

This seemed to be a point that Curtis took great umbridge with.  I cited an example where a user actually argued with me that his on-array thin provisioning was really great.  Then he told me that he keeps an extra 40 to 60 percent of his capacity in reserve just in case a margin call happens.  That made me shout:  THEN WHY ARE WE USING THIS TECHNOLOGY TO BEGIN WITH?

I shouldn’t have shouted.  That was bad form. 

And maybe I shouldn’t have referenced idiotic car commercials like the one that Mercedes is airing right now about its smart driver alert systems that inform a driver when he is drifting into the adjacent lane or coming up too quickly on a stopped vehicle or falling asleep at the wheel.  To my way of thinking, this feature is for folks who probably are not really capable of driving safely to begin with.  Metaphorically, I said that alot of on-array value-add functionality is being bought by people who are probably not really capable of managing storage properly.  That seems to have triggered Curtis and many of his commenters.

In the final analysis, I wasn’t asking whether we should have dedupe or thin provisioning or tiering.  I was asking why these functions need to be embedded on an array controller, where they jack up cost, rather than provided as software services — perhaps in a virtualization layer or as a part of systems managed storage as we have done for years in the mainframe world.

Isn't this a question worth asking?

My presentation style may have been too noisy for Curtis — too “bad boy” — but the question is a valid one and goes directly to the cost of storage, which is obscene.

The presentation went on and I won’t belabor it here.  But I will say that buying brand name rigs for the brand name makes little sense.  I continue to argue that lots of value add software creates hard to manage stovepipes of storage that are cost accelerators from a TCO perspective.  I continue to rail against the pundits and analysts who cheer on each new evolution of the shiny new features of most storage rigs without considering what they are doing to the costs of the rigs for companies that buy them.  If that makes me “a contrarian” with “a bad boy act” who “doesn’t care whether people agree with [me],” then that’s just fine with me.

Curtis is entitled to his view.  I don’t read his blog routinely, but his backup book is very good. 

The only thing I really wanted to correct was an assertion made in one of the comments:  “He made disparaging comments about the vendors and then specifically slandered and mocked ReiJane Huai.”  I did disparage vendors, if by disparage you mean I disagreed with their marketing around value-add software.  I did not slander and/or mock Rei, who I greatly admire.  I imitated his delightful accent and quoted his comment made in a video interview at my site regarding deduplication (“a waste management system for backup”) and on-array thin provisioning (“a big scam designed to let hardware vendors charge a lot more money for the same disk”).  But, for the record, I did so out of reverance for his authenticity and honesty, and not in any way to slander him.  I apologize to Rei if this came across as anything else.

FalconStor and DataCore and many others are trying to deliver services like thin provisioning across all spindles through a virtualization layer, which is much more cost effective and intelligent than doing it on individual array controllers.  Xiotech gets this point and eschews on most of its rigs the insertion of value-add functionality that could be done by its ecosystem partners, united via Web Services REST protocols.  Makes sense to me.

Containing storage costs comes from the smarter building of storage infrastructure, not "smart storage"

Hope that this clarifies the context of Curtis’ remarks and provides a fair presentation of the facts for future discussion.

{ 6 comments… read them below or add one }

wcpreston November 9, 2010 at 9:57 pm

Given that I basically said in my blog that you were full of crap ;), I appreciate the overall civil tone of this response.

I got the main point of your presentation. It’s the same main point you made in the Infosmack podcast. I do believe there’s another side to that particular argument, but I chose not to take that one on.

I chose instead to take on one of your sub-points, that thin-provisioning is to storage management what Lane Assist is to Mercedes. I understand the context in which it was said, but when you said “thin provisioning is for people too stupid to manage storage,” there is no context that softens that position. You didn’t say anything that sounded like “oh, but if it was in a software solution, I’d be fine with it.” I got the same impression about what you said about dedupe.

BTW, as long as I’m on dedupe, Sepaton has never offered a software option. The price to which you refer ($2500/TB) was the cost to upgrade pre-dedupe Sepaton customers to dedupe Sepaton customers. Sepaton has never had a software-only anything. They have always been exactly what you say you hate, some storage (right now HDS storage) attached to a controller that runs value-added software that turns that disk into a virtual tape library and now a deduping virtual tape library. You cannot buy the controller w/o the disk, and you cannot buy the disk w/o the controller, and you definitely cannot buy the software w/o the controller.

To my knowledge, only FalconStor offers a software-only target dedupe appliance. IBM does offer a gateway model that you can plug into any supported storage, but Data Domain, Exagrid, Quantum, NEC, Greenbytes, & Avamar are all controller/storage combos.

As to “slander[ing] and mock[ing] ReiJane Huai,” your words were that he was about to be indicted. Since they have just begun an investigation, have no findings, and to my knowledge the entity doing the investigation has no indictment powers, saying that he is about to be indicted is factually incorrect. Since it was spoken (as opposed to written) disparages his character, and is untrue, that is the very definition of slander. And one person’s imitation of a delightful accent is another person’s mocking. I saw more than one person look at each other when you did that with a face like “did he just make fun of an asian guy’s accent on stage?”

Off to board a plane. Gotta go.

Administrator November 10, 2010 at 9:19 am

Thanks for the feedback, Curtis…I think.

It was apparent to me that your blog was tinged with disdain. As I have said to you in the past, I have zero interest in endeavoring to persuade you to any position. My response was civil because, despite my bombastic presentation style, I pride myself on a certain degree of civility in personal matters. I am in fact striving to disagree without being disagreeable.

If I was unclear about my preference for software-based storage management functionality, I apologize of course. To regular readers of this blog, my articles and my presentations, it has always been about off-controller software solutions versus on hardware solutions. I thought I had made that clear in the presentation foils in this case, and on stage. I am sorry if I offended you with my remark about stupid people who need “smart” cars or, metaphorically, “smart” storage.

In too many cases, I see folks buying into on-array thin provisioning because they are essentially lazy. Not always, but often. Paying through the nose for a value-add feature because you prefer to be lazy is always an option, of course. But if functionality is really valid — that is, if it has a real benefit to offer — wouldn’t it just make sense to implement that functionality in a scalable way — in externalized software — so that you realize economies of scale by applying it to all storage capacity?

As for Sepaton’s software only option, I was told that this option was available by spokespersons for the company at the rate specified in the slide. I don’t know whether I should take your word for it, or theirs. I am thinking their word is probably the choice I would make until they clarify this matter for me.

On the matter of Reijane Huai, if I said indicted, I misspoke. I am not familiar with the particulars of the events that led to his resignation, only that I hold the man in enormous respect and would never diminish his contribution to the industry or his delightful sense of humor. I do not consider imitation of his syntax and accent to be mocking. I have a habit of imitating every accent I hear, and his is delightful. I envy the way that he succinctly and with great humor characterizes the truth. (Rei, if I have offended you, I apologize. My intention was not to slander or mock at all — and mocking/slandering is defined by its intent, not by the act itself.) In fact, he would agree with my position on the above.

wcpreston November 10, 2010 at 6:47 pm

Since the dictionary says that disdain means unworthy of consideration, I think it’s a bit strong for the situation. If that were the case, I wouldn’t have written the blog entry in the first place. It is true that we disagree on many points, and it is true that I strongly disagree with the manner in which you present your points. But I don’t believe my (or your) personal feelings should interfere with our ability to have a civil discuss over an individual point.

Your preference for software-based smarts was NOT unclear. It was the main point of your presentation. Also, you did not offend me. I thought you offended some of the audience by calling them stupid. I still maintain that it appeared that you were at war with the very concept of thin provisioning, NOT just thin provisioning built into an array. If that was not the case, I will acquiesce.

As to the Sepaton thing, I can only ask you to consider that this is my area of specialty (backups and related devices). I was on the technical advisory board for Sepaton at one point, helped write their VTL and dedupe best-practices guides, and have been in the middle of many pricing negotiations involving that product. If anyone outside of Sepaton knows that product better than I do, I don’t know who it would be. I can understand that you would want to take their word over mine, but what I’m telling you is that if you heard them say they had a software-only option of ANY kind, either they mispoke, or you misunderstood. The DeltaStor software runs ONLY on Sepaton hardware.

It is the epitome of what you do not like (value-added software that runs only on specialty hardware) . The only company (to my knowledge) that does VTL & dedupe the way you like (as software you can run on any box with any disk) is FalconStor. If you’re cool with a separate box that you have to buy from the same vendor, then IBM also does it. Feel free to double-check what I’m saying against any source you would like.

Finally, I’d like to just state a counter argument to your overall point of software on the array. Each approach has its advantages. Your approach has the advantage of being able to somewhat easily change your underlying disk vendor because you’re not tied to it via the value-added software issue. The main advantage to the monolithic approach is the single throat to choke. In addition, there is a significant amount of effort that goes into making sure part A works with part B. When you go with the monolithic approach, you are paying someone else to do that work for you. If you go with your approach, you have to do that work — every time you change a piece of hardware. It’s amazing the rippling effects minor changes can have. This is why many customers prefer the monolithic approach, even if it marries them to that vendor for the life of that hardware.

Administrator November 11, 2010 at 4:41 pm

Thanks for the additional feedback, kind sir. I certainly yield to your encyclopedic knowledge of VTLs and dedupe. In fact, my inquiry to Sepaton returned a confirmation of your technical point. They do not sell software only, but offer an appliance both as a company and in conjunction with HDS. The $2500 per TB number is what they charge to extend the software cover to additional capacity once you buy their appliance.

I am glad that you pointed this out, since it was unclear from my original conversations with product management there. It does establish, however, the same baseline that I was seeking for comparison. $2500 for the software versus a hardware plus software solution that jacks up the price of disk drives by 100x.

I know that some folks go monolithic for the reasons you indicate. I think that this is wrong-headed as I will endeavor to model in an upcoming post.

Thanks for caring.

wcpreston November 11, 2010 at 5:14 pm

So you’re going to make the point that they charge you $2500 per TB to turn that TB into 10+ TB (assuming a 10:1 dedupe ratio, which is a conservative estimate)? What’s so wrong with that?

Administrator November 15, 2010 at 1:47 pm

Not at all, Mr. P. I never said anything bad about deduplication per se, except that it is very difficult to derive anything like the ratios promised by the vendors. If you need the technology and you have $2500 per TB to spend, go for it. Embedded on an array, however, you are looking at an extreme mark-up on commodity disk — 100x.

This point, again, is obvious I thought from the slides presented in SF and reposted here.

Previous post:

Next post: