Business Challenges
DirecTV was faced with a legacy VM based catalog, search and notification service. With the system at its maximum, scaling options were limited. Worst case scenario plans included fixed over-capacity scaling to cover any unexpected demand. Consequently, system idle time became unnecessarily wasteful. Further, costly twice yearly operations to scale the system up for the game season and to scale it down afterwards were standard practice.
Transformation
Integrated Digital Solutions collaborated to recommend and build a completely transformed system that would leverage the cutting-edge serverless technologies provided by AWS to overcome the business challenges.
Cost reduction
84% reduction in cost by employing an elastically scaled solution and eliminating customer provisioned load balancers in favor of AWS Elastic Load Balancing
Efficiency
Migrated system to AWS standardizing it with DirecTV’s other workloads. Eliminated twice yearly scale out and scale in activities and the associated functional and load testing
Reliability
AWS Elastic Load Balancing provides highly available load balancing which eliminates single points of failure. It also integrates with AWS auto scaling to automatically replace failed application.
Scalability
AWS auto scaling ensures that the system scales automatically to meet user demand and avoids the cost of provisioning.
WHEN UPTIME IS THE MVP: FINDING THE WINNING STRATEGY FOR A FLAWLESS, HIGH-DEMAND SERVICE.
Business Challenge Story
DirecTV built a system that served 300,000+ loyal subscribers every Sunday during the NFL season. This system provided catalog, and search and notification services which required stringent SLAs to ensure consistent and reliable uptime This system worked as built but over time, difficulties became apparent.
The as-built-system idle time was costing an estimated $400k annually. This idle time fluctuated based on the ability to accurately predict capacity. Annual planning sessions to predict capacity were prone to error and typically included over-estimations to prevent system load problems. Once determinations were made about capacity, there was no capability to dynamically scale during the season and the system was kept at specified capacity until turn down at the conclusion of the season.
SUNDAY MVPS, MONDAY SAVINGS: HOW IDS’S AUTO SCALING SCHEDULE TRANSFORMED NFL HIGHLIGHTS PRODUCTION.
Transformation Story
In a competitive market, IDS presented AT&T with an AWS-centric system that would overcome business challenges as well as reduce operating costs. Consider the following conclusions presented by IDS:
During the development of an enhanced solution for the production of highlights for DirecTV’s flagship NFL Sunday Ticket product, IDS became aware that the catalog, search, and notifications systems were statically provisioned at the start of the season, scaled for worst case demand, and then scaled in at the end of the season.
IDS quickly realized that due to the strictly periodic and scheduled structure of NFL Sunday Ticket, compute capacity was idle nearly 85% of the time.
IDS recommended auto scaling with a schedule that scaled the system out on Sundays and scaled it in on Mondays, coupled with the ability to automatically scale to meet unexpected demand from Monday through Saturday.
Utilizing “scheduled actions” meant that the weekly scaling activities could be limited to just the NFL season.
A Perfect Solution
The IDS presented solution was accepted and built over a two month period. IDS created a pre-prod environment where the applications were deployed to ec2 instances, AMIs were created, and auto-scaling and load balancing policies were tuned. The resulting solution was deployed to production where is has been serving 300,000+ customers since 2017.
Result Story
RESULTS AND BENEFIT
The resultant IDS presented, developed and deployed solution gave an 84% cost reduction over the previous system. These savings were achieved by: scheduled scaling to ensure capacity availability for users immediately without lag elastic scaling to ensure that the system can handle unexpected surges in traffic automatic replacement of failed application servers to ensure availability of content for users and the migration of the system to AWS reducing ops burden (no need to know about and maintain multiple cloud providers).