You could say it's been a busy 18 months for Srividhya (Vidhya) Narayanan, Director of the OnCue Cloud Platform at Verizon.
In September, the company rolled out go90, its mobile-first content platform. The new product offering represents a major milestone in the company's big bet on the mobile content opportunity.
The "Over the Top" (OTT) content trend that's catapulted companies like Netflix into powerful positions in the entertainment industry represents huge revenue opportunities for the nation's largest wireless carrier, who industry watchers are following closely given their massive wireless installed base and traction with the millennial audience. As ad spend shifts from traditional TV to digital, the company sees a tremendous opportunity for mobile advertising.
Verizon's leap into the space with go90 all happened at a dizzying pace for Narayanan, who was a director on the OnCue team at Intel, which Verizon acquired in 2014 as the core intellectual property that the entire go90 service would eventually be built around. go90 is a social entertainment platform, built for the mobile-first generation, offering access to live and on demand video wrapped in a social experience.
Now aboard Verizon via the Intel OnCue acquisition, Narayanan joined the team creating a software architecture allowing the rapid integration of Verizon’s newly acquired assets for its mobile content strategy, and supporting continuous integration and deployment as the company marched towards the launch of go90. The expectation of the go90 platform was high reliability, high availability, and ease of use for Verizon’s wireless customers and other carriers.
It became obvious to Narayanan and the Verizon team that the only way to achieve this was by isolating “edge services” from any internal systems that could potentially have failures. So the go90 team set out to build a microservices architecture, and began evaluating the programming languages and technologies that would support their journey.
“Microservices -- much more than being just a buzzword -- represent a common sense way of developing and deploying a highly distributed system,” said Narayanan. “Building services that care for a specific domain and its use cases means planning, development and deployment of these components are done independently and more frequently without the need of turning everything into lock step. go90 was only possible with the right microservices architecture.”
Srividhya NarayananDirector of the OnCue Cloud Platform at Verizon
For Narayanan and her team, the microservices patterns Verizon wanted to follow for its go90 platform quickly led them to a short-list of requirements for language and technology choices. At the top of the team’s list:
Scala and Play stood out as strong technology choices for various reasons. The connection to the JVM was an attractive factor. The fact that the Verizon engineer’s existing Java skills translated to Scala was a major plus. But the real intriguing part for the team was Scala’s principles of pure functions that helped keep services immutable - patterns that were critical to horizontal scale and the resilience required to provide a seamless user experience for mobile customers.
Narayanan and her team saw the combination of Scala, Play and the Lightbend Reactive Platform as the path to better isolation by removing single points of failure and contention. They saw an opportunity for good principles and practices at the language layer that could lead to truly share-nothing designs, asynchronous message passing, and splitting up the work into small composable chunks for elastic scalability.
With a microservices architecture, Narayan knew that one of the greatest challenges was keeping service-to-service communication cost extremely minimal. A single client request would often turn into several calls to multiple services, or several database requests. Also, many of these calls had to be done in parallel. So Play’s principal design of concurrent request handling was an absolute necessity for these Verizon use cases. And Play’s non-blocking design also helped Narayan and her team to effectively use cloud resources (instead of having threads locked).
Verizon had its hands full with the combination of hiring and ramping up a team of developers, taking a product to market at a fast pace - all the while with a new language and platform. While there was strong internal consensus that their architecture and development goals were well supported by the Lightbend Reactive Platform, the team also realized they needed help along the way. So they partnered with Lightbend for support.
“That partnership was the best way to get a team ramped up for all the challenges,” said Narayanan. Lightbend helped us with everything from customized training for the development team to the availability of knowledgeable consultants working alongside our developers to rapid patches as needed -- we were able to get the platform built reliably and on-schedule.”
Using Scala and Play, the go90 team built a microservices architecture that fully isolated “edge services” from the back-end systems. Narayanan describes edge services as basically a set of Play-based services with no dependencies on the back-end. These edge services broker the calls between the client devices and the back-end, and are designed to be extremely fast and resilient. If the back-end dies, the services still respond to the client application and continue going.
This isolation not only solves a lot of availability problems, but optimizes the customer experience. Rather than the client brokering call requests to back-end systems, the business logic is done on the server side, and edge services serve up the data to the client with optimal speed.
The Go90 team also has adopted a system of solving common service problems as libraries so they can be shared by all microservices across the different engineering teams. The service-to-service communication, DB access, authentication, logging, monitoring are good cases for libraries that the team maintains. These libraries adopt common patterns for error handling or circuit breakers to ensure the application is able to stay resilient for error cases. Scala’s composition model offers advantages that have come in handy in designing circuit breaking patterns and ease of implementation. Some of these also have been open sourced at http://github.com/oncue
“All of this would not have been possible without the right architecture and principles,” said Narayanan. “I take a lot of pride in the Web Services platform and how we increased agility and speed in our development cycle.”
Despite adding more than 100 microservices and performing more than 10 deployments per week, client endpoints are not exposed to systems that could fail under the high rate of product evolution. This has allowed Verizon to have a simpler implementation on the application side. The architecture also has allowed Verizon’s engineering team to move faster, because work on new client functionality is completely decoupled from the back-end, and all that’s required is plugging new deployments into the edge services.
Now that the team feels like experts on all things Scala and Play, they have been venturing into other Scala frameworks as well. Whether running Apache Kafka within Mesos or Spark on Mesos for machine learning, analytics or advertisement features, the team has been embracing these new technologies at a rapid pace.
Inspired by this story? Contact us to learn more about what Lightbend can do for your organization.