Fred Moore is a pretty savvy guy. I read his annual Storage Manifesto to get the infographics he has collected from the analyst community and his own insights after an XX-year long career in storage. He gave a talk recently at a storage symposium and our fly on the wall provides this summary. (Fred, if you are reading, please feel free to correct any of these notes.)
The storage industry grew 5% in 2007. This was noteworthy considering that it has been steadily shrinking since 2000. In 2000, industry revenues were $31B versus $18B in 2003. Activity on storage startups remains anemic. In 2000, there were 2000 bona fide storage startups. This fell to 37 in 2002, fell further in 2004 to 16, and fell even further to 2 in 2006. In 2007, there were 9 or 10 startups. Still a low level.
Associated with this small number is the lack of genuine new ideas. Since 2000, the only new ideas have been de-duplication and flash memory. All other ideas animating developments in storage have been long known, for example, storage resource planning.
In 2006, 99 storage companies were acquired. Continuing the entity consolidation underway. In 2007, there were only 6 disk drive makers left down from over 100 in the late 1980’s. In contract, there were 9 storage IPOs in 2007.
Recently, a new marketing trend has been underway wherein resellers and integrators invite “C” level individuals to meet with them and chat about top of mind issues. What follows is there top of mind thoughts and information.
- 70% of their servers have been virtualized by the end of 2007.
- Important needs (things they want to buy and implement) include (in order of importance): data classification, data prioritization, data lifecycle management, storage security, data protection, storage virtualization (including virtual tape in non-mainframe environments).
- Companies are moving away from an emphasis on CAPEX to OPEX. However, they still love a bargain sales price even if on-going costs are higher than they should be.
Fred to the industry, “When are we (in industry) ever going to begin to focus on where the money’s going (OPEX)?” Fred sees that as the area for opportunity.
State of the Industry comments
In 2007, the first 1 TB hard drives were shipped, a notable achievement.
Energy consumption concerns have emerged and they are responsible for a renewed interest in tape as well as in lower HDD energy consumption. Western Digital announced a 520 GB per square inch aerial density which supports 640 GB per platter disk drives. By stacking 4 to 5 in an HAD you get 3TB HDDs.
Fred predicts that as usual this astounding achievement will be forgotten in the rush to the next aerial density milestone. However 3TB per HDD must give pause to consider the implications.
In tape, LTO with encryption is the big story. Also derivative technologies, e.g., flash, hybrid storage devices.
The storage industry has now finally entered the 4th era. The first era was mainframe and centralized data centers. The 2nd was distributed architectures. The 3rd was network centric architectures. The 4th is information centric architectures (and thinking).
Fred asked the audience to consider how long they could survive without their data system? This period should determine strategies for data protection, etc.
Demand for storage was over 12 exabytes (EB) in 2007 (compared with 6.4 billion people). 92% of the data is digital. UNIX, Linux, and Windows-based represent over 90% of this consumption. 80% of all existing data sits on tape. 8% of the new data is analog generating 1.8PB of paper-based storage in 2007.
The following averages represent what CIOs reported:
Data Classification Percent of Data Stored Mission Critical 15% Vital 20% Sensitive 25% Non critical 40%
For small to medium sized enterprises (SMB), 36% of their applications were mission critical. 2/3 were not. What data protection strategy should you use for applications that were not mission critical? An unanswered question.
Data Sources (creation)
Enterprise (data centers) 10 to 15% Desktop 40 to 50% Personal 10 to 15% Consumer electronics (ex. iPOD) 20%
Key end user storage initiatives in 2008:
- Server and storage consolidation to lower cost (OPEX) and complexity.
- Virtualize servers and storage
- Grow network storage capacity
Top storage vendor initiatives in 2008
- Tape and drives deploy encryption features across the board
- Disk and flash hybrids
- Thin provisioning on non-mainframes
- Data classification and tiered storage
- Tier Zero storage (solid state)
- Power and cooling: green mode for disk, striping usually works against minimizing energy consumption
- Energy consumption is out of control with end user bills rising 20% to 30% per year.
- Energy providers still get a free ride. No pressure or cost to them for reducing consumption.
Key Disk Demand Drives in 2008
- Disk taking applications from tape and optical: backup and recovery, data protection, security
- Personal appliances: iPODS, cameras, camcorders, backup/recovery
- 20% of IT workforce is mobile.
- New disk applications gain momentum (more workloads opening up for disk storage)
- Internet applications everywhere: about 1.8 billion users worldwide
- There are 4.3 billion internet addresses. IPv6 will fix the looming shortage. Supports 16 billion billion addresses.
- Compliance, fixed content growing 60% per annum.
- Two disk classes are emerging: performance, high capacity
- Storage management complexity and value rising fastest of any factor (far in excess of 60% per year)
- $ / GB falling 40% per year
- Access density increasing less than 10% per year.
- Drive capacity increasing over 60% per year.
- Disk performance advancing less than 10% per year.
- Non-mainframe shop storage utilization is falling and is projected to fall over the next 5 to 10 years.
Access Density Discontinuity
Access density = IOPS divided by GB. Device capacity growing 30% to 40% per year. Device performance increasing 5% per year. Therefore, capacity remains king however disk performance does not scale with capacity increases.
The market is now Tier 3 primarily (long term archive). VTL adds significant value for tape storage vendors. A hot product category is integrated disk and tape. For tape to remain healthy/viable, aerial density must continue to exceed disk. Removable disk and MAID are yet another competitive challenge being directed at tape. With the demise of STK, who will be the leader in the tape industry? It is still a $4.2B industry in 2008. Tape cartridge capacity exceeds the capacity of hard disk drives and the gap is widening.
Tape Best Practices (for end users)
- Select the right data for tape storage: fixed, compliance, green.
- Use automation whenever possible.
- Deploy VTL.
- Deploy encryption.
- Ascertain vendor commitment to tape.
- Discern who has the “vision” for tape
Form Factor Outlook
2 ½ inch HDDs will be the floor for HDDs. Flash will take whatever market space is below, e.g., 1” HDDs.
Price Data (all data includes all components needed like disk controllers)
Solid state $1,000 to $4,000 per GB Enterprise disk $18 to $40 per GB Midrange disk $10 to $25 per GB Optical Blue Ray $10 to $30 per GB Economy disk $2 to $12 per GB Automated Tape $0.30 to $2.50 per G; $0.20 on mainframe
Energy Facts and Figures
- One hour of sunshine meets the worlds entire energy needs for one year.
- Energy consumed by data centers worldwide doubled from 2000 to 2005.
- Data centers consumed 1.5% or US electrical consumption in 2006.
- Worldwide electricity use: US 40%, Europe 25%, Japan 12%
- Worldwide 2 billion people have no access to electricity (the electrical grid). It would be cheaper to deploy solar than to connect these 2 billion people to the electricity grid.
- Developing countries consume 30% of electricity generated. (Some double counting with statistic cited above.)
Storage and Energy consumption
- Hardware costs go down with functional improvements.
- Energy costs go up without improvements (even assuming steady state in equipment, etc.)
Power usage in Data Centers
Chillers 33% UPS 18% AC 9% IT equipment 30% Within IT equipment Servers 35% Storage 30% Networks 23% Other 12%
Note: Storage overall effect is 30% times 30% equals 9% of the total energy cost.
I am not sure how Fred gets some of these numbers or what his full thinking is with regard to some of his recommendation, so I guess an interview is in order. I will get right on it.
For now, I feel like my world is a more orderly place because I have Moore’s take on the state of storage. Fred operates a consultancy called Horison Information Strategies. He does some work for vendors, but he has a “do no harm to the consumer” focus I much admire.