UK’s Online Gaming Leader Bets Big on Reactive to Drive Personalization
Real-Time Customer Data Becoming New Battleground Within Gambling Industry
For some organizations, “Going Reactive” is a consideration for the future. For industry leaders, Reactive systems have already enabled major competitive advantage.
The United Kingdom legalized online gambling 10 years ago. Although the market for online “punting” is still very young, it’s growing at a sharp clip. In the first five years the annual market reached £1.48bn, accounting firm KPMG estimated. By 2014, the UK Gambling Commission confirmed market growth to £2.44bn.
With hundreds of gaming operators fighting for slices of this action, one of the UK’s oldest offline gaming operators -- William Hill -- managed not only to enter the nascent online game, it quickly secured an astounding 16% of market share. By 2014, William Hill reported £399m in online gambling revenue, up 175% from the previous year. Now they’re number one in online gaming.
So what role does technology play for the growth strategy of the UK’s largest online gaming operator?
Doubling Down on New Personalization Features and Functionality
In the online gambling industry, the basic betting services offered by operators are marginally different - and the odds themselves have become so commoditized that many large operators globally even offer free APIs for live odds. Operators compete fiercely for differentiation with user-facing features and functionality. William Hill attributes a large part of its marketshare to simply delivering a better customer experience than other gambling operators in the UK.
We can’t predict the future, but we know if we are not in control of our data, we’ll never be in a position to innovate. Understanding the structure of our data and how we can leverage it to deliver highly personalized content to our users is the next major opportunity for revenue growth.Patrick Di LoretoR&D Engineering Lead, William Hill
Di Loreto and William Hill have continued to explore ways to grow marketshare on the basis of what the customer wants. They believe their most interesting opportunities are tied to personalization - assigning logical reasoning to customer behavior, and presenting personalized data on the basis of machine learning and intelligent predictions.
For William Hill, “personalization” takes on an even greater importance with the rise of “In-Play” markets - where users can bet on live games, with new betting options presented in real-time (e.g. "will Sharapova win the next point in a tennis match?") In-Play represents the ultimate challenges for personalization. How to make users aware of new betting options that match sports and teams they are interested in is the challenge. William Hill needed computing infrastructure that can instantly draw correlations between user actions on their site, other sites they visit, betting propositions they look at and act on, what other similar players do under similar circumstances, and lots of other reasoning (deductive reasoning, inductive reasoning and abductive reasoning), all in a blink of an eye.
Many of our new, instant campaigns need to feature personalized recommendations that may only be valid for thirty seconds, and all that must perform perfectly during special events and namely on Saturday afternoons, when we regularly face up to 100x traffic peaks.Patrick Di LoretoR&D Engineering Lead, William Hill
A Winning Hand of “New Stack” Technologies Fuel William Hill’s Data Streaming and Path to More Personalization
To capture, correlate and understand in real-time every customer interaction leveraging the power of machine learning, Di Loreto realized his team had to re-imagine its application architecture to deliver on the realities of all of the data in motion and logic required to exploit it. As core technologies for data streaming–like Spark, Cassandra, and Kafka–came into view, it became very obvious that the world with Java, Hibernate and Spring was not going to be able to deliver the right foundation for bringing more personalization opportunities to the business.
Our journey to Reactive started with embracing core functional programming concepts and continues with a more responsive and resilient system. Really we saw that a change in how we thought of system architectures was going to be a prerequisite to ‘Going Reactive, and after a crash course in Scala we decided to use it to prototype Omnia. We felt it would let us handle all the data wrangling and logic required to manage the data streams on our users. It was clear that core elements of functional programming—monoids, observable subgroups, location transparency—were must-haves.Patrick Di LoretoR&D Engineering Lead, William Hill
As William Hill embraced functional programming, they quickly arrived at the combination of Scala and Akka as the backbone of their application infrastructure. With Scala handling the most difficult issues of concurrency, and Akka Actors doing the heavy lifting around location transparency and negotiating with other key distributed frameworks - William Hill is quickly building a set of tools at the data layer, to create a foundation for long term success in personalization features and functionality.
Getting to Know William Hill’s Home-Grown Data Streaming Frameworks Written on Top of Scala, Akka, Spark, Kafka, Cassandra, and Mesos
- Omnia—The primary production data platform, collecting user information from all activities on the website, as well as online and offline events related to the customer.
- NeoCortex—a home-built Spark framework that allows developers to write applications that are simple to read, and spares them from the complexity of distributed computing and multithreading.
- Chronos—the Play Framework-based application that collects user activity both inside and outside the website and connects to all applications internally and externally (including Facebook and Twitter).
- Fates—another custom-built framework that correlates specific activity streams from Chronos and makes them configurable as timelines of customer events.
- Hermes—the service layer that makes all the data managed by Omnia available to B2B and B2C applications.
A “New Stack” for a Customer Data-Driven Business
As William Hill builds the new “Reactive Stack” for its customer data-driven business, it has been an aggressive adopter of complimentary distributed frameworks that play very nicely with the key Reactive tenets of Scala and Akka, and make the company’s data platform Reactive all the way through. Each technology assumes a different role that compliments the others.
- Functional by design, brings core characteristics for concurrency, and doing things in a declarative fashion.
- Brings concurrency to all systems that William Hill builds (where Java concurrency was overly complex, Scala has programming abstractions that make it easier to write concurrent code).
- Di Loreto calls Scala a “prerequisite” to going Reactive; from his point of view, the observable and Reactive elements to William Hill's’ system start with monoids, and Scala is the key.
- Along with Scala, serves as the backbone of Omnia and other core data frameworks at William Hill; allows the team to program the software and not care about where the components live physically.
- Handles key aspects to clustering and sharding functionality that allows information in the Omnia platform to be equally distributed inside of a cluster.
- Heavy use of Akka Persistence and Actors connecting other frameworks brings resilience, fault-tolerance and removes any single points of failure to William Hill's data platform.
- The data computation engine that handles all of the analytics of user information.
- Works very fluidly with Cassandra (the primary datastore in William Hill’s data streaming architecture).
- Core to real-time processing, logical reasoning, graphing and other heavy lifting related to correlating different datasets related to William Hill users.
- Distributed database, manages storage in a Reactive manner and creates the timelines that get passed to Spark for logical reasoning (very solid integration with Spark for CQRS systems, with mature connectors).
- Brings high availability and embraces the view that failure is a normal part of a system, to ensure that no single failures can bring the system down.
- Handles “horizontal scale” extremely reliably / efficiently.
Mesos and Docker
- All of William Hills’ new data architecture is written as microservices that are contained in Docker containers and run on top of Mesos.
- Mesos manages tens of Docker containers - launching, managing, provisioning resources.
- Brings elasticity to the cluster, CPUs, memory, I/O and other infrastructure, and adds portability of infrastructure, convenient packaging and deployment of services with isolation from other services.
The Results - Unshakable Resilience at the Speed of Real-Time Big Data
For William Hill, time is truly of the essence. Knowing the latest details for the most popular events–and being prepared to handle massive spikes in traffic during events–is a key differentiator both in their home UK market and worldwide.
Maintaining resilience in the face of 100x peaks in traffic is something that William Hill deals with on a weekly basis. For example, Mondays and Fridays are typically “slow days”. Tuesdays and Thursdays frequently see 20x the traffic of slow days, and Wednesdays (when Champions League is on) can reach 50x the traffic. However, it gets interesting on Saturdays in the UK between 3 - 5pm, when William Hill regularly sees a 100x increase compared to the previous day, and drops off again significantly on Sundays.
Given the millions of customers that William Hill serves, handling this type of load required a highly performant framework that could be distributed across different systems and handle millions of parallel requests. William Hill utilizes Spark on top of Akka clusters for exactly this: Akka serves as a transparent processing framework for all requests, enabling Spark to process the real-time data, using Kafka and Cassandra for messaging and storage.
Simply, without Omnia using Akka and Spark to handle distribution and speed, none of these services would have been possible to launch due to our real-time requirements and extreme peaks in load.Patrick Di LoretoR&D Engineering Lead, William Hill
A Reactive Foundation for Future Microservices
Patrick and his team launched Omnia on July 21, 2015 in order to prepare for their peak season, which goes from September to May each year. Along with the July launch comes two services that William Hill are counting on to engage their clients like never before:
- Recommended “Bet Suggestions”: Omnia’s recommendation engine works in a similar way to Amazon’s media suggestions, with the major difference being that compared to a discount on a book, William Hill’s “Bet Suggestions” are fleeting and may only last seconds. The need to concurrently handle millions of recommendations with very fast lifetimes would have been impossible without Akka actors and Spark.
- Gamification: William Hill wants to explore whether gamifying the experience outside betting and gambling leads to higher engagement and have designed a feature that rewards customers for performing specific actions in a short amount of time, sometimes lasting only seconds. These activities include personalized challenges specific to the event or sport the customers wants. Not only is it challenging to provide these services, but the ability to concurrently process it all during traffic peaks is also needed. Millions of customers are performing actions and William Hill needs to be able to provide relevant challenges and also reward the customers in real-time during busy hours.
Since the launch, Omnia has been available to VIP clients, so the real test of Omnia comes on August 8th, when William Hill is planning for a pre-season peak during the UK’s Premiership League.
To prove that Omnia can handle large audiences, at 3pm on the Premiership League opening day William Hill will test the recommendation engine by sending out 500,000 personalized offers, most with life-spans of less than 5 seconds, to users on their platform.
The most important feature for Omnia right now is to cope with large numbers of simultaneous, real-time offers. This is the first chance for us to see how Omnia reacts to this new type of services, where previously we couldn’t even consider without these new technologies powering it all.Patrick Di LoretoR&D Engineering Lead, William Hill
Needless to say, William Hill is investing heavily in the technologies powering their Omnia platform and has instituted an innovation team in the tech startup haven of Shoreditch in London. The goal: explore new concepts and features, mostly by leveraging Omnia’s reactive capability, to make brand new products and services that continually embrace Reactive technologies.
Inspired by this story? Contact us to learn more about what Lightbend can do for your organization.