Why Flash is the Fastest Growing Form of Data Storage
Flash memory is used every day in thumb drives and hand-held smart devices. But it’s no longer relegated to consumer electronics. According to a June 2016 survey from IT consultancy 451 Research, almost 90 percent of organizations have some form of flash-based storage installed in their data centers. All-flash storage systems are also becoming increasingly standard for transactional applications.
The findings don’t surprise Andy Walls, CTO and chief architect, Flash Systems, IBM. “In more than 35 years at IBM, I’ve seen storage change in amazing ways,” he says. When Walls joined IBM in 1981, HDDs were the gold standard of data storage. Since then, he notes, they’ve had hyperaccelerated growth. “In many ways, HDDs are one of the most important inventions in all of human history. The improvement—the bits per inch, the scaling—has been absolutely mindboggling.”
While HDD technology is durable—in the past 30 years, scientists and engineers seemed to circumvent physics to continue scaling HDDs—Walls says its history of remarkable innovation is slowing.
“A disk drive is a mechanical device,” he explains. “Spin media and magnetic heads can read the bits off of that media. Physical restraints limit how fast it can spin, so you really can’t get an average response time of less than a few milliseconds, even with all of the tricks we’ve learned.”
Flash Drivers
Today, more organizations find that flash storage systems provide better performance and efficiency in data centers than HDDs do.
In 2008, when flash (formally known as NAND flash memory) started to get into its hyperaccelerated phase, it became the core storage technology of consumer devices. “Usage increased to the point where a lot of both competitors and supply existed,” Walls says. “Suppliers were working on reducing the cost.”
Flash grew so inexpensive that developers realized it could be used in data centers. Because the cost wasn’t as low as HDD storage, it was used primarily for niche or fringe applications that were latency-sensitive and could afford the premium. One early use was automatic tiering. “We could put flash and HDDs together in the same storage enclosure and determine which data went where,” Walls says. “That way, your average response time and I/O operations per second improved, yet the cost didn’t increase much.” Because the total cost of ownership is less for flash than it is for HDDs, IBM has seen wholesale adoption of flash storage for many clients.
Meanwhile, consumer demand continued to drive flash production. Manufacturers kept making flash devices smaller and cheaper. This drove more usage in data center applications. The performance aspects of flash helped drive the adoption rate. But that performance also allowed for data reduction techniques that further reduced the effective cost of flash-based storage. These techniques are particularly slow with HDDs because additional metadata may also be read from slow HDDs. Flash, however, can absorb the additional metadata fetches and still provide excellent performance.
Using Flash
One of the main uses of flash storage is what Walls calls a hyperconverged network, where an organization can put storage in servers that also contain storage. Then, software such as key-value store or Apache Cassandra is used to scale out both the compute and storage. “This new phenomenon has been accelerated by flash,” Walls says. “It’s become cheap enough and has lower cooling requirements than HDDs. IBM Power Systems* servers play into this well because a rich assortment of in-server flash products can be put into servers and scaled in a similar way.”
The second main use for flash-based storage in the enterprise is the external storage market. Organizations can now buy all-flash arrays that have similar total cost of ownership to HDD arrays. An example is the IBM FlashSystem* A9000, which incorporates compression and deduplication. “These all-flash arrays can provide extremely good performance—better than what you’d get with an HDD system, with data reduction that can be as high as 5x, depending on the type of data,” Walls says. This kind of data reduction allows clients to move workloads from HDDs to flash.
Walls’ team is enabling the consolidation of storage. Previously, if an organization wanted to stand up a new application—for example, a new database application using an Oracle or IBM database—it had to buy storage. “Virtual partitioning and server virtualization have changed that model so that you don’t do it anymore for servers,” Walls explains. “You can partition some virtual machines to that new workload.”
This idea applies to storage as well. Instead of having storage in every server, or stand-up external storage for each application, you can now consolidate multiple workloads onto one flash array. Clients are starting to view their flash arrays as a pool of storage they carve out at capacity as customers need it. Flash allows them to operate as an internal cloud.
Growing Flash
Business workloads currently drive flash storage growth. Flash allows organizations to efficiently consolidate virtual desktop infrastructure (VDI) workloads, and virtual server instances (VSIs). All-flash storage arrays can allow a 10-to-1 and even 12-to-1 deduplication for these workloads. For instance, flash is particularly helpful in avoiding boot storms in VDI environments: when everyone turns on their workstations at the same time. “This isn’t a problem with flash at all,” Walls notes. “All-flash arrays can easily handle the performance requirements of these events.”
“Clients that have had difficulty meeting their service-level agreements on transaction workloads using Oracle, SQL or other databases can bring in a FlashSystem 900 and see immediate and significant latency improvement.”
—Andy Walls, CTO and chief architect, Flash Systems, IBM
Organizations can also make use of flash using “tier-zero” storage products, such as the IBM FlashSystem 900, which can be used with many IBM i and Power Systems workloads. Walls describes the FlashSystem 900 as a fast application acceleration storage device that attaches to fiber channel storage-area networks. It doesn’t have rich data services, such as replication or deduplication, but it does allow clients to quickly and simply improve performance.
“Clients that have had difficulty meeting their service-level agreements on transaction workloads using Oracle, SQL or other databases can bring in a FlashSystem 900 and see immediate and significant latency improvement,” Walls explains. “Sure, the database could be improved. But having database administrators spend time on that may not be the best use of their time when flash storage provides immediate payback.”
The End of Hard Disks?
With all of the benefits flash offers large data, will HDDs soon be extinct? Walls doesn’t think so. Near-line HDDs have low cost per raw gigabyte even though performance isn’t improving. However, HDDs are still cheaper for colder data that’s not accessed often, if at all.
But active data, which includes storage for Oracle, SAP and VSI workloads, is increasingly transitioning from all HDDs, or having some HDDs and flash, to all flash.
In short, Walls adds, “I don’t think you’ll see your traditional transactional workloads on HDDs for much longer. HDDs will become more of an online archival repository—an object store where you store your colder data. Analytics can still be run on it, but it’s not accessed often.
“You don’t need flash for that,” he continues. “The latency isn’t as important—it doesn’t need to be available online all of the time. But you do need it available for compliance reasons or historical reasons. And often, this is a lot of data. Storing this kind of data is absolutely geared to and perfect for an HDD.”
Even with the boom in flash storage, HDDs will continue to play a role in enterprises. “When new technology comes along, we think it will eliminate the other technologies,” Walls says. Sometimes data storage technologies go the way of the floppy disk, but some companies continue to archive cold data on tape.
“The advent of HDDs pushed tape to its ideal place, which is archival and backup,” Walls explains. Flash might do the same for HDDs. However, to quickly access data, flash is becoming the standard for storage. With its high-performance capabilities and innovations in development, it’s a smart choice—and not just for smartphones.