De-Dupe as a File System Feature
I knew it was coming. ZFS apparently has it — or shortly will, per this post.
Read the follow on comments as they raise several key issues involving crypto in particular. I was surprised to see little or no discussion of the ramifications of this technique for file sharing in an environment that isn’t completely ZFS enabled.
In any case, score another one for ZFS!
I wonder whether the generally queasy experience many have had with earlier forms of compression and encryption in the file system will impede adoption of this technology…
November 6th, 2009 at 1:05 pm
Just a continuation of the trend to off-load data efficiencies into the storage layer. Anything that can have a broad application will just end up being the ambient condition of data storage: RAID, dedupe, compression, encryption, cloning. Think of this as off-loading these processes to the storage so the server(s) (virtualized or not) can devote cycles to running applications.
The challenge to-date is getting what some have referred to as “file system” features into legacy storage arrays. The people that have built a file system from scratch (although, NetApp would contend that ZFS was not built from scratch but from ONTAP), have the best shot at getting these features off-loaded to the storage array. The big 3-letter companies are finding it increasingly tough to keep bolting on these features and use them in combination with one another in a practical, production environment.
Unfortunately, as these features clear the first hurdle of at least making it on the spec sheet for a product, the onus shifts to the consumer to figure out who is selling features that run only on overhead projectors and who is selling features that would work in their production environments. From Jeff’s post you get a lot of “I cannot offer any absolute guidance” to “pretty fast” to “will add overhead.” That’s not a dig. It’s absolutely truthful on Jeff’s part because now the heavy lifting needs to start and that’s including sizers, best practice guides, application qualification.