Ticketfly wanted to enter a new and previously untapped market. To do that, they needed a solid foundation on which to build the systems that would support this new offering. The foundation needed to be massively scalable, reliable and highly performing. Ticketfly found that foundation in Lightbend technologies.

About Ticketfly

Ticketfly is an integrated ticketing and digital marketing platform for event promoters and venues of any size, from standing-room-only clubs to festival grounds and arenas with reserved seating. The Ticketfly platform provides a full suite of integrated ticketing, social marketing, email and analytics tools that streamline operations and increase ticket sales.

Ticketfly was founded in 2008 by Andrew Dreskin, the founder and CEO of TicketWeb, the first company to sell tickets online in 1995, and Dan Teree, the former president of TicketWeb. Our team builds cutting-edge technology to improve the live event experience. The San Francisco Business Times recently named Ticketfly the fastest-growing technology company in the area.

The Problem

Ticketfly was well known as the fastest General Admission ticketing company on the Internet, but they were not content to sit on their laurels; GA ticketing only comprises about 20% of the online ticketing market. The other 80% of the market comprises of the “Reserved Seating” space, which allows customers to pick and search for seats in the best areas of stadiums and events.

Reserved Seating Engines are by their nature, very difficult to build, with many factors to consider:

• There is a lot of rapidly changing and coordinated state, which must be accounted for, and this is inherently difficult to scale.
• Traffic patterns have extreme bursts and can be unpredictable. When a popular event has its onsale, the moment tickets become available to the general public, there can be tens of thousands of simultaneous requests contending for the same resources. Factor in that any number of other artists or sporting events could be on sale at the same time and the required scaling parameters are almost unfathomable.
• The underlying seat and venue availability is changing constantly, due in large part to the load on the system from users.
• Finding the best seats, searching for small clusters in a graph, rating those clusters against other clusters while those clusters are constantly changing is a real algorithmic challenge. Doing this in a distributed system where data can live on different nodes, and factoring performance and resiliency into the mix makes the task that much harder.
• Ticket scalpers have become more sophisticated; they are writing Bots that can send so many simultaneous requests to online ticketing engines that combined with normal on-sale traffic, it actually looks like a distributed denial of service attack.

With all of these challenges to face, clearly Ticketfly needed to make their infrastructure decisions very carefully.

The Solution

Andrew Headrick, Ticketfly’s Platform Architect was tasked with leading the team that would build the reserved engine. Given the challenges Andrew faced from a business-logic standpoint, he needed the infrastructure he would use to be scalable and resilient with minimal effort.

He noted that a tactic commonly used by other reserved seating systems is to push all the complex state management into the database layer – essentially coding an application in database stored procedures. This however leads to a very inflexible system with tightly coupled operations and components, not to mention very expensive database licenses.

As he assessed the task, Andrew reached instinctively for two technologies that he’d previously used on other projects; two components from Lightbend’s technology platform as the basis of the engine: Scala and Akka. Scala is inspired by the long tradition of functional programming, which makes it easy to avoid sharing mutable state so that computations can be readily distributed across cores on a multicore server, and across servers in a datacenter. This makes Scala an especially good match for modern multicore CPUs and distributed cloud-computing workloads that require concurrency and parallelism. Akka, on the other hand is a toolkit and runtime for building highly concurrent, distributed, and fault tolerant event-driven applications on the JVM.

The overall solution lay in a CQRS Event Sourcing architecture that would thereby allow Andrew to bring the business logic up into the application layer as opposed to pushing it down into the database. He modeled everything much of the system as finite state machines (FSM); seats, indexes, check out processes, etc.

Lightbend’s Akka middleware provides a stable and easy to use Actor library. Actors by their definition of being an address with behavior and state makes them and excellent platform for building state machines. With the additional tools of Actor's become/unbecome methods, Scala's partial function composition and Akka's FSM trait modeling state machines is that much easier. Furthermore by encapsulating business logic and state management inside actors makes many of the common problems associated with concurrent and distributed programming “just go away”.

An FSM can be described as a set of relations of the form:

State(S) x Event(E) -> Actions (A), State(S')

These relations are interpreted as meaning:

If we are in state S and the event E occurs, we should perform the actions A and make a transition to the state S'.

Specifically in Ticketfly’s world, one critical component the FSM models is seating inventory for a specific event; this means that the seating graph is stored in RAM across a clustered, distributed system. To enable this, indexes separate and listen for state transition events to keep things up to date as follows:

• Request comes in.
• Message is fired to the FSM monitoring indices.
• Reply back with “seats available”.
• Check out process sends individual messages to each individual seat to change state to “allocated”.

Clearly, the goal is to do these operations concurrently and non-blocking, so that entire blocks of seats or sections are not locked. One feature that enables the massive level of concurrency required are Akka Futures - Andrew enthusiastically mentioned that Futures are “better than sliced bread!”.

Andrew and a small, efficient team of developers built the reserved engine, and today it is comprised of about 30,000 lines of Scala code. Andrew states that the system “has very good performance without having to try very hard to achieve it”. One way this is demonstrated is that finding and allocating seats under load happens in less than 20ms.

A side benefit of the reserved engine is that it allows access to other inventory systems too, which in turn allows other Ticketfly systems to scale.

In an ecosystem where missteps and outages are instantly Tweeted for all the world to see, systems going down is a very bad thing indeed; Ticketfly feels safe and confident that they made the right choice with Lightbend.

Inspired by this story? Contact us to learn more about what Lightbend can do for your organization.

GET IN TOUCH