Big data-generating applications are geared to flood data centers and edge infrastructure (as well as personal devices) over the next few years. Applications such as higher resolution, higher frame rate and higher dynamic range video, IoT applications and numerous big data and artificial intelligence activities will tax our digital storage resources as well as the network bandwidth required to move this data around. Fortunately, increased processing power will help us manage this onslaught of data and keep storage and bandwidth resources under control. This article will look at several storage and bandwidth efficiency offerings using advanced processing power.
Perhaps the best current example of what processing power can do is the use of High Efficiency Video Coding HEVC (H.265, MPEG-H Part 2) compression for reducing the size of video content. HEVC is much more compact for a given image resolution than H.264 (conventional MPEG) video. HEVC utilizes significant processing power on the encoding side in order to implement lossy compression of the video content, where the compression is focused on parts of the images that will be least noticed by viewers.
On the decoding side, much less processing power is needed and thus the computation requirements at the receiver are not significant. As a result, between 20-40% compression is possible. This technology has enabled 4K video delivery using conventional video delivery channels, such as cable, which is now offered by many video streaming services. HEVC can be used for 8K or higher video as well.
At the 2018 NAB show Pixspan was showing their lastest development in loss-less video compression technology. Their WAN Xpress enabled 3X data transfer speeds, with real-time (that is on-the-fly) frame by frame loss-less compression encoding. The technology was demonstrated over a Signiant network. They noted that noisy grainy film is the hardest to compress, but with other content they were able to achieve up to 60% compression.
Pixspan also demonstrated their VDrive with a PXZ file generated from Maya rendering using GPUs. These PXZ files can be 50-90% loss-less compressed from the original file. While stored as a PXZ file it can be reconstructed as a full DPX file. These files can be stored locally to save net storage space and lower delivery bandwidth. The company was showing this technology for content delivery but perhaps it could be the basis of a loss-less compressed video storage as well as delivery of content in a post-production environment that allowed real-time decompression of the content.
At the 2018 GPU Tech Summit Disney Imagineering gave a demonstration of real time cinematic 4K quality video rendering at 60 Hz using 8 GPUs. Rendering rather than storing content could be an additional way to save on some storage requirements and as the cost of processing goes down rendering could become a realistic solution for high resolution content creation on the fly and might even allow creating real-time special effects during production.
In addition to real-time creation of content, faster processing can be applied during the ingest of content to analyze data as it is stored. This allows the creation of advanced metadata that would make searching and managing the ingested content much easier. Many storage vendors are starting to implement machine learning (ML) and other AI techniques to help them monitor and manage content (such as recent Dell/EMC announcements) and it is a logical step to apply AI to create metadata for stored content during ingest.
Increased processing power is making it possible to save on data storage capacity and content delivery bandwidth. These advances will help us manage the coming data tsunami. In addition, processing data to extract metadata during content ingest will help improve our access to and management of stored content.