Wait … what?
I know that sounds off-the-wall, but its fun to see the similarities in the story arcs. In the movie, the plot evolves from misunderstanding between humans and dragons, to understanding between humans and dragons, to people and dragons living in harmony. Digital Transformation includes a similar story arc. Applications and storage technology exist on either side of a wide gulf. Application administrators are typically not storage experts. Similarly, storage experts struggle to define seamless integrations with application requirements. If we could achieve greater understanding between the two worlds, we could move towards application-centric infrastructure where applications connect to a fluid data infrastructure, where storage technologies meet the application’s needs according to policy and with minimal cost.
It’s a great story. Or it could be a great story if we could wrestle the plot to that happily ever after. We’ve made progress, but more is needed. Automatically harmonizing application requirements and storage technologies according to policy remains a difficult problem to solve.
Why?
First, Know Your Dragons
Today’s modern storage industry produces a vast array of different data storage technologies. Each storage technology meets different parts of a wide spectrum of application requirements. To create a policy-driven selection algorithm, matching technology to requirements, we need a map of that landscape of technologies and requirements.
CNCF Storage Landscape is such a map. Meant to facilitate discussion of storage technologies in the areas of public and private clouds, this well-organized paper hints at the difficulty of understanding and mapping the variety of storage platforms used in production applications.
Mapping a complex space is difficult when it’s a static picture. It gets worse when the picture is changing rapidly. Digital transformation drives changing application requirements. These drive evolution in storage technology. It’s hard to define a mapping from requirements to technology when both ends of that mapping are changing. And so an automated policy-driven data infrastructure remains elusive.
And there’s a bigger problem. Applications themselves are transforming architecturally to become AI/ML applications. Soon, all companies from enterprises to SMBs will find they are big data companies. Companies will need to keep all their data to maximize the potential of those AI/ML-enabled applications to deliver business value. The already exponential data growth rate will accelerate. Not only will data grow but the kinds of data will grow. And this data must be available, accessible, and with cost minimized when it’s not being accessed. This is another dimension of evolutionary pressure on storage.
It’s hard to map the space, and it’s made harder due to transformative change driving increased data growth and changing data performance requirements.
A Better Behaved Dragon is still a Dragon
Still, the industry has taken meaningful steps toward an application-centric data infrastructure.
First, the emergence of software-defined storage marked a pivotal moment in the evolution toward application-centric infrastructure. Storage pooling allowed heterogeneous storage systems to be managed uniformly. Different storage systems within a pool combine logically to support application requirements. Thin provisioning in these pools made capacity planning easier, but it also enabled fine grained auto-tiering algorithms. These algorithms follow application access patterns and allowed software-driven storage architectures to balance application data between cheaper storage and faster storage, dynamically, without applications being the wiser. The data could be performant when needed, but stored on cheaper technology when not needed to minimize cost. This was a real policy-driven innovation, but humans were still needed to match storage policy to application requirements.
Virtualization platforms began to evolve policy-driven storage abstractions. VMware’s VVOL specification provides a rich framework for the expression of application requirements and storage features. Microsoft’s System Center Virtual Machine Manager (SCVMM) integrates SMI-S and a similar Storage Management Protocol (SMP) to enable policy-driven management of storage systems. OpenStack’s Cinder interface integrates storage into private cloud infrastructure. The storage systems provide a menu of options, and applications declare their requirements. The platform mechanically brings the two together. As a bonus, this might actually satisfy the administrator’s expectations most of the time.
These are all examples of better behaved storage serving the needs of applications. But the overall system could do a better job automating the process of matching applications to storage technologies meeting their requirements, lessening the need for human intervention to settle disharmony.
How People and Dragons Live Together
A paradigm shift in application architecture has interesting implications for storage.
Container technology isolates application functionality into logical namespaces within the host operating systems. At some point, application data must be stored on persistent volumes. The industry is defining a way to connect containerized applications to persistent storage. Drivers for Docker persistent volumes and Kubernetes Container Storage Interface (CSI) abstract storage to deliver it to containers.
Sound familiar? It should. These are not yet policy-based systems like VMware VVOL, but they look like the early stages of that evolution. Systems like Rook enable a declarative language and take steps toward a requirements-based integrated system.
There’s another possible path of evolution. First, note that software-defined storage converts storage into a software program. In this way, storage becomes a kind of application. This is a powerful notion. It means that the same technologies that are driving radical improvements in applications—AI, ML, containers, cloud—can also drive improvements in storage itself. Storage becomes AI/ML enabled, massively scalable, nonlocal … it transforms entirely into a fluid infrastructure that responds dynamically and intelligently to application needs.
Think about it. We are fascinated by stories in which AI automatically predicts customer needs around holidays, updates inventory, and reduces cost when demand subsides. Why should storage be any different? We can turn storage itself into an application, where AI automatically predicts peak periods of I/O demand, moves data among different technologies to increase performance, and then moves data to cheaper technologies when demand subsides.
Now you don’t need a prescriptive map to define the policy. The relationship between applications and storage becomes an organic, living thing, not the mechanized algorithm by which all interactions between applications and storage dragons are prescribed.
That would be a happy ever after indeed. I’m not sure my daughter would watch it over and over again, but I bet it would be a hit in data centers around the world.
Read More on Storage Dragons
Read more:
