How Switzerland’s Largest Telecom Provider Shipped a Major New Data Pipelining Service In Less Than a Year With Lightbend Reactive Platform
At Lightbend, we see that every company today is in the software business. Software is indeed “eating the world” so the world’s most innovative companies are viewing their developers and data assets as the keys to new revenue growth and operational efficiency. And the languages and platforms they use are the critical infrastructure powering their journey.
Swisscom is the leading mobile service provider in Switzerland, with a network that Connect Magazine deemed the nation’s best in a 2015 test. For Swisscom, analyzing the vast array of equipment across its cellular network has always been a requirement for maintaining a high level of service for customers. But in March 2015, Michal Piórkowski was tasked to lead a “Mobility Insights” team that broadened the outlook on Swisscom’s mobile data in search of all advantages that could be derived for not just network intelligence and optimization.
Lightbend was excited to join Piórkowski and his team as they set their sights on a very specific opportunity they saw in Swisscom’s data assets: to enrich urban planning with Switzerland’s largest set of mobile user data. Swisscom realized that where census data tended to be static, outdated and limited, anonymized mobile data from its network (more than 60 percent of Switzerland’s mobile users) could support a wide range of use cases where comprehensive, real-time data on urban populations unlocked huge opportunities:
- Accurate Population Density Mapping: Mobile equipment can contribute anonymously to a real-time view of the head count of a group of people in the population of a given location, which can help derive a density map of the area. This data can be refined to the density maps with street-level granularity: this already supersedes the census surveys of old. And the relevance of this information is much better than a census since it can be obtained continuously, in real-time, as opposed to the manual headcount usually collected by administrations on a yearly basis (at best).
- City Planning: A second insight that can start from this stepping stone is an origin-to-destination matrix that can be used to figure out where people come from and where they are going. This is valuable, for example, for civil engineers working on the particular tools needed to travel through a location, but also for traffic planning, transportation planning, and ultimately as building blocks for smart city planning.
- Intelligent Traffic Routing: Another output from Swisscom’s massive data set included ideas on how public transport could differ during sporting events and whether some train or bus lines should have special times. These could include which roads it is more important to work on, from the point of view of the city or from the Stadium’s administrator.
From Concept to Commercially Available Assets in Less than Nine Months
The use of Lightbend support was a fantastic asset — having access to the creators of the language let us do our best to make sure that Swisscom stays close to the cutting edge of technology in the Scala and Spark ecosystem.
With Swisscom’s vision set to create a data analytics platform that could deliver detailed analysis at nationwide scale (and sold to urban planners across key use cases)––Piórkowski and the Mobility Insights Team set out to build the platform that could deliver these new data services.
In a mere nine month period, the Mobility Insights team achieved a set of commercial assets with its first paying customers. Piórkowski attributes much of the astonishing speed of developer innovation at Swisscom in building this new product platform to some key technology choices.
The volume of data, the requirement for real-time computation, and the associated performance requirements immediately led the Mobility Insights Team to Apache Spark at the big data processing layer. The platform also needed to match the stringent quality of service expectations (high availability, reliability, resilience under peak usage, etc.) that are table stakes in telecommunications, which led Swisscom to the Scala language and Lightbend Reactive Platform (as well as a Lightbend customer support subscription). According to Piórkowski, following are some of the key benefits of the Scala language that fueled new product innovation at Swisscom:
- Interoperability with Java Libraries and Legacy Code. “We enjoyed the great interoperability with Java libraries and legacy Java code. Moreover, writing code for scientific computation in the modern world needs to leverage the existing Scala ecosystem (such as the Breeze library), as well as the Java landmark libraries in the field (such as Apache Commons Math), but also letting the user plug in native code alternatives in the back end for performance (something achieved through Spark through netlib-java). This seamless interface with the Java world avoided a lot of time spent reimplementing basic computations in an optimal manner.”
- Strong Typing. “Scala is a strongly typed language, which helped guard against common bugs in unit mangling, and solved the problem through type decorations.”
- The Power of the Scala-graph Library for large cell graphs. “The Scala-graph open source library — originally created at and now maintained by EPFL — was a key asset in the construction, representation, modification, and easy computation on a large cell graph. Moreover, it scales well enough to represent all the paths of interest that we are testing, each numbering in the hundreds of cells.”
- Domain Specific Languages. “One of the biggest advantages of Scala for us is that it leverages advanced syntax (operators) and structure to ease defining domain specific languages — we found we could very quickly express the transformations that we are interested in our graph computations and for the recognition of users on our various paths.”
- Tapping into Spark Streaming and Kafka. “As a consequence of the scale of our data analysis, we distributed the matching of users against paths of interest, using Spark Streaming and Kafka as an event delivery engine. In terms of algorithmics, crafting an interesting graph of cells representing a path of interest was done using scala-graph, but the actual matching had to switch to techniques based on locality-sensitive hashing, distributing an index of selected short paths across a cluster of machines. This technique, implemented by hand from techniques born in the 90s, let us do approximate matching against user histories.”
- Lightbend Project Success Subscription (PSS). “The use of Lightbend support was a fantastic asset — having access to the creators of the language let us do our best to make sure that Swisscom stays close to the cutting edge of technology in the Scala and Spark ecosystem.”
Swisscom’s platform leverages many other complementary technologies in real-time data pipelining and analytics, including Kafka, Cassandra, HDFS. For more information about Swisscom's Smart City initiatives, read this follow-up Q/A blog post with Francois Garillot, Big Data Architect at Swisscom, and visit their dedicated Smart City microsite.
Inspired by this story? Contact us to learn more about what Lightbend can do for your organization.
GET IN TOUCH
