As your digital video workflows grow in scope, your underlying storage strategy must adapt. The days of buying one tier of storage for editing and one tier for archive are quickly coming to an end due to rapidly evolving asset reuse, globally distributed workflows and on-demand delivery requirements.
So, how do you define your storage strategy to create better, more cost-effective workflows? What are the characteristics of each tier of storage needed? And, what are the variables to consider when defining how much of a specific storage tier you need? It all starts with understanding the rapidly evolving characteristics of digital video and how these map to the type of storage best suited for your particular needs.
What Are the Top Storage Challenges of Video Content?
1. Increasing File Sizes
1080P workflows are quickly turning into 4K workflows with 8K workflows starting to take hold and 16K cameras looming on the horizon. Therefore, both the size of the image (the resolution) and the size of the resulting files are growing. In addition, uncompressed workflows are now being requested, resulting in multi-TB sized files.
2. Aspects of Creation & Consumption
The rise of video sharing social networks, video on-demand (VOD) services, broadband and mobile devices have changed the way content is produced and consumed in several ways, including:
- Consumers now view content when they have time, on the device that’s most convenient to them
- Resolutions range from mobile (720, 1080, 2K) to 4K with 8K TVs now on the market
- You now have to create various versions, dimensions, resolutions for the exact same piece of content
3. Protection & Access
With video being repurposed and reused, nothing is thrown away, and the need to protect content and keep it accessible is critical. Here are some of the factors that intensify the need for protection and access:
- Video content has intrinsic value well beyond the initial use
- All project files, source footage and produced content need to remain instantly accessible
- On-demand workflows are straining workflows based on tape, it takes too long to recall assets
What are the Requirements for Video Storage?
- Speed: What is the speed required to perform your particular task? If you are editing 8K footage in an uncompressed format, you are going to need really fast storage. If you need playout functionality playing back compressed formats over the web, you don’t need as much speed. The goal is to match the speed to the task.
- Length of File Use: How long are files going to be used for their immediate purpose? How long do they have to stay on a particular tier of storage?
- Repurposing: When is a specific file going to be reused? Is it within 6 months?
- Archiving: If you know when a file will be repurposed, you can develop your archive strategy. For instance, if you are not going to use a file within 6 months, perhaps it’s time to move that file to a secondary or tertiary tier of storage.
Once we look at our workflows and file use characteristics through the lens of these requirements, we can start to determine what capacities we need for each specific tier of storage to complete our task.
Now let’s take a look at the five tiers of storage available for video production workflows.
What are the 5 Tiers of Video Data Storage?
1. Ultra-Fast NVMe
Ultra-fast NVMe is similar to RAM. This is the type of storage needed for 8K or 16K uncompressed workflows since it is faster than SSDs. You can get content off the drive quickly due to low latency and do it with less drives than used in a traditional RAID array. However, networking requirements are massive, along with the resulting compute and overall price tag. This isn’t a cheap solution which is why you buy the amount needed for your specific task.
2. Ultra-Fast SAN/NAS Architecture
This is your more traditional SAN and NAS environment needed for high-speed editing platforms. They are fast and reliable and optimized for video workflows with large capacity drives and arrays with a multitude of connectivity options (10 Gig, 40 Gig, Fibre…). However, networking upgrades may be required. There are high costs of ownership due to power/maintenance, and performance degrades as they near capacity. These factors are behind the need to eventually offload files to different tiers of storage.
4. Local Archival
If your content has a shelf life of 3–6 months then you may want to consider putting it on a NAS or filer. They are relatively low cost (vs NVMe and Ultrafast SAN/NAS) with Petabyte-level capacity. However, they have limited scalability beyond multiple Petabytes, are often not accessible over the web, and have long rebuild times in the case of a sector or drive failure. In addition, support and maintenance fees in subsequent years can often exceed the original purchase price.
3. NAS & Filer
For on-premises and local archive, you have two options. LTO (tape) or Object Storage (HDD- based). The benefits of LTO are low cost per TB with virtually unlimited tape capacities. However, you are locked into a tape format, maintenance is often an issue, and if the files are stored across different tapes, it can take a long time to recall assets. The benefits of object storage, like DataCore Swarm, are also the low cost per TB with a cloud-like low maintenance experience and dense capacities. However, object-based storage requires available data center footprint and (like LTO) an HSM or data mover, such as DataCore FileFly, to plug into certain file-system-based workflows. For most organizations, dealing with new video production workflows and on-demand requirements, object storage is the tier of storage with the most rapid rate of growth.
5. Cloud Archival
The cloud also employs both LTO and object storage technologies to enable archive services. The benefits include zero maintenance with unlimited capacity and instant deployment. The benefits, however, come at costs (capacity, ingress/egress, and API calls) that compound over time and are sometimes difficult to predict.
Which Storage Tier is Best for My Requirements?
Depending on your requirements, you can match the appropriate type of storage. This is illustrated in the chart above. The circles in the chart above represent the amount of time that files are going to sit on each tier of storage, based upon the requirement. For instance, if speed is the primary requirement then files will sit on NVMe or Ultra-Fast NAS/SAN. However, if archive is the primary requirement, the file will most likely be on your local archive or cloud archive tier. This table is another tool you can use to help you map your current workflows to the associated tier of storage.
Which Storage Workflow is Best for Video Production?
Similar to the previous matrix, the table above shows what storage tier is ideal for a specific workflow. For instance, for production you would most often use NVMe or Fast NAS/SAN. For video-on-demand (VOD) or over-the-top (OTT) enablement, you would most often use a local archive or the cloud. You can also use NAS for VOD and OTT; however, you would need to add additional layers of infrastructure like a web server. For archiving projects or masters, a local archive or the cloud are the clear choices.
Conclusion
The number of tiers of storage you deploy and their resulting capacity is all going to come down to your budget and the requirements determined by a needs analysis. If you are struggling with editing 8K or 16K uncompressed files, then you should take a look at NVMe. On the other hand, if you are struggling with VOD/OTT enablement or keeping archives instantly accessible, you should take a look at object storage, like DataCore Swarm, or the cloud.
We hope that the information covered in this blog and the different tools presented help you define your storage strategy. As always, we are here to help. If you have questions or would like to schedule a demo, contact us today.
