WHEN UPTIME IS THE MVP: FINDING THE WINNING STRATEGY FOR A FLAWLESS, HIGH-DEMAND SERVICE.
Business Challenge Story
DirecTV built a system that served 300,000+ loyal subscribers every Sunday during the NFL season. This system provided catalog, and search and notification services which required stringent SLAs to ensure consistent and reliable uptime This system worked as built but over time, difficulties became apparent.
The as-built-system idle time was costing an estimated $400k annually. This idle time fluctuated based on the ability to accurately predict capacity. Annual planning sessions to predict capacity were prone to error and typically included over-estimations to prevent system load problems. Once determinations were made about capacity, there was no capability to dynamically scale during the season and the system was kept at specified capacity until turn down at the conclusion of the season.
SUNDAY MVPS, MONDAY SAVINGS: HOW IDS’S AUTO SCALING SCHEDULE TRANSFORMED NFL HIGHLIGHTS PRODUCTION.
In a competitive market, IDS presented AT&T with an AWS-centric system that would overcome business challenges as well as reduce operating costs. Consider the following conclusions presented by IDS:
During the development of an enhanced solution for the production of highlights for DirecTV’s flagship NFL Sunday Ticket product, IDS became aware that the catalog, search, and notifications systems were statically provisioned at the start of the season, scaled for worst case demand, and then scaled in at the end of the season.
IDS quickly realized that due to the strictly periodic and scheduled structure of NFL Sunday Ticket, compute capacity was idle nearly 85% of the time.
IDS recommended auto scaling with a schedule that scaled the system out on Sundays and scaled it in on Mondays, coupled with the ability to automatically scale to meet unexpected demand from Monday through Saturday.
Utilizing “scheduled actions” meant that the weekly scaling activities could be limited to just the NFL season.
A Perfect Solution
The IDS presented solution was accepted and built over a two month period. IDS created a pre-prod environment where the applications were deployed to ec2 instances, AMIs were created, and auto-scaling and load balancing policies were tuned. The resulting solution was deployed to production where is has been serving 300,000+ customers since 2017.
RESULTS AND BENEFIT
The resultant IDS presented, developed and deployed solution gave an 84% cost reduction over the previous system. These savings were achieved by: scheduled scaling to ensure capacity availability for users immediately without lag elastic scaling to ensure that the system can handle unexpected surges in traffic automatic replacement of failed application servers to ensure availability of content for users and the migration of the system to AWS reducing ops burden (no need to know about and maintain multiple cloud providers).