AT&T / DirecTV Video On-Demand

AT&T already boasted one of the most expansive VOD catalogs in the industry, but various significant acquisitions signaled a critical VOD library expansion. Additionally, this new content, as well as the existing back catalog, needed to be processed in a way that was optimized for the new and upcoming consumer platforms. The challenge was to maintain the high standard of excellence in customer experience, while introducing new content and new platforms within a noticeably short delivery window.

There were a few major concerns to consider. For one, each VOD title took an average of ten hours to process with a substantial associated cost. How long would it take to reprocess the entire back catalog? Could the new catalog of assets even be processed in time to highlight the acquisitions? Secondly, the error and subsequent correction workflow would need to be improved as improperly processed and missing titles were an ongoing concern. Third, there was no clear long-term resourcing and maintenance path for the existing system as overhead costs would continue to rise along with the number of titles processed. Lastly, questions were being asked, “Could the existing system be used to process media content across all AT&T properties? Or would other technologies be required for each individual entity?”

Soon it was clear that without significant modifications and upgrades, the existing VOD processing platform would simply not be up to the task. A new and advanced approach would be needed.

To architect and achieve this next-generation VOD processing solution, AT&T turned to their trusted innovation and development partners at Integrated Digital Solutions (IDS), an AWS Partner Network member. IDS worked directly with AT&T to understand the problems faced with the current system and provide innovative solutions.

Evaluation of the existing VOD processing workflow revealed an on-premises solution that utilized a mixture of third-party and open source-based encoding, packaging, and video QC tools. Not only was the system expensive due to hardware and licensing costs, it was also slow and not scalable. This legacy system suffered from the typical scaling problem from which most on-premises solutions suffer in having an unpredictable workload resulting in either lost time and missed SLAs due to under provisioning or lost money due to over provisioning. Furthermore, the use of third-party solutions meant that, without a large capital expense, the system was inflexible and limited to only the encoding and packaging formats and QC validations supported by these third-party tools.

IDS suspected that given AWS’ unparalleled, integrated ecosystem of serverless technologies, AWS technologies would be a natural fit and a perfect response to AT&T stated business concerns. However, this would be a significant and substantial change in direction. This novel approach would also mean abandoning invested systems that had previously provided business value. Was this approach truly better than transforming the existing on-premises system?

This impressive proof of concept allowed AT&T to rapidly decide and proceed with a new cloud-based VOD workflow transformation. As the proof of concept progressed into production stages further unforeseen advantages became apparent. AT&T realized that their development could iterate on theories much more quickly and inexpensively than previously. This rapid failure/success approach gave rise to innovation and energy that was previously blocked.

This model allowed for complete customer ownership, control and understanding of their cloud processing system. The system is completely unique to the customer.

After understanding AT&T’s requirements and the problems with the existing solution, the IDS team developed a suggested architecture. Some components were a given. For instance, content would be uploaded to s3 for processing and output to s3 for delivery. Other components, such as the orchestration mechanism and the ephemeral storage, allowed for some flexibility. The IDS team started the development by building a proof-of-concept system. This proof-of-concept system was developed in a modular fashion that allowed IDS to swap out the various components allowing them to understand and characterize both the costs and behavior of different architectural choices.

While the AWS Elemental MediaConvert solution may seem like the ideal solution, AT&T’s long-term goals meant that they required complete control over the entire encoding and packaging process. This in turn meant that a custom solution was necessary. The ability to orchestrate a mixture of lightweight, composable lambda functions and longer running tasks on elastically scalable compute resources was seen as crucial to the success of the solution. The fact that AWS provides this set of features through its AWS Step Functions, AWS Lambda, and AWS Batch services and does so with minimal effort and little boilerplate made AWS the leading choice.

After trialing various orchestration and ephemeral storage solutions, IDS found that the fastest and lowest cost solution was achieved through the use of a set of Lambda functions which are triggered when content is uploaded and are orchestrated by AWS Step Functions. These Lambdas invoke AWS Batch jobs to perform processing functions with I/O and/or execution time constraints not compatible with Lambda. For example, while AWS Lambda supports associating EFS file systems with functions, it was found that attaching FSx for Lustre file systems to containers managed by AWS Batch provided both a lower cost and a lower processing time.

By moving the VOD processing workflow from an on-premises system to a cloud-based servers solution, AT&T reduced the per-title cost of processing by 90%. Additionally, the time to process each title was reduced from an average of 10 hours to less than an hour. These cost and time savings allowed expansion of the VOD catalog while meeting internal SLAs and ensuring that the existing catalog was optimized for the new AT&T TV product offering.

Beyond the concrete benefits of reduced cost and processing time, the optimization of AT&T’s VOD workflow promotes innovation. By minimizing cost and processing time, AT&T’s product and engineering teams are able to perform experiments with alternative encoding, DRM and packaging solutions with the confidence that they can deploy the best technology to customers across the breadth and depth of the sizable VOD catalog. This agility has already allowed adoption of CMAF packaging and recognition of additional cost savings by replacing DRM systems with alternates.

The optimization of AT&T’s VOD workflow also resulted in some radical and unexpected cost savings. Like much of the rest of the industry, AT&T employed “Just In Time” or JIT packaging of its VOD assets. JIT packaging provides support for the multiple outputs required by different video players by reading a single common source and packaging it the requested format on the fly. It does so to minimize storage costs, which it trades for the requirement of always-on compute resources, resulting in higher than necessary costs, operational complexity, and performance impacts. The low cost, high performance, and high reliability of s3 coupled with the low cost and low processing time of the new VOD workflow meant that AT&T could eliminate “Just In Time” or JIT packaging by packaging each asset in all of the required formats. In doing so, AT&T was able to eliminate an entire cost center with a sizable monthly cost for compute resources, load balancers, and egress traffic.