Today Lightbend announced that it has been selected by IBM to lead Big Data University's core curriculum in the areas of Scala, Reactive Programming and Apache Spark. IBM launched Big Data University in 2011 as an industry initiative to spread "big data literacy" to more than one million data scientists. The expansion to the curriculum brings new opportunities for developers seeking to master the latest technologies in Big Data (in a recent StackOverflow survey, Scala and Apache Spark were tied for first as the "Top Paying Tech in the U.S.").
Big Data has become the killer app for functional programming (FP) and functional languages like Scala. The emphasis on immutability improves robustness, and data pipelines are naturally modeled and implemented using collections (like lists and maps) with composable operations. The phrase "Fast Data" captures the range of new systems and approaches, which balance various tradeoffs to deliver timely, cost-efficient data processing, as well as higher developer productivity.
For years, Lightbend has been at the forefront of innovation in Fast Data, working with companies like Samsung (case study), Verizon (case study), William Hill (case study) and many others to achieve the benefits of Reactive systems in Fast Data architectures. Enterprises leverage Scala, Akka, and Play Framework (the Lightbend Reactive Platform) to reduce the time gap between data arrival and value extraction. Lightbend is also a leading support provider for Apache Spark users (Spark is implemented in Scala), and an author of the Reactive Streams specification that solves Fast Data's most challenging backpressure requirements.
Fast Data architectures require reliable data ingestion, flexible storage and query options, and sophisticated analytics tools. The components that meet these requirements must also be Reactive, meaning that they scale up and down with demand, they are resilient against failures that are inevitable in large distributed systems, they always respond to service requests even if failures limit the ability to deliver services, and they are driven by messages or events from the world around them.
"We see Fast Data and the focus on speed -- more so than data volumes -- being the principal driver for the new class of Reactive applications that must work with streaming data," said Jamie Allen, Senior Director of Global Services at Lightbend. "Scala brings major advantages for building out these types of systems. We're honored to work with IBM to support this education to further their massive investment of resources in the data science community to bring complementary frameworks like Apache Spark to market."
New courses led by Lightbend in IBM's Big Data University will focus on enabling data scientists -- particularly those currently using Python or R -- to leverage Scala and its ecosystem of complementary tools to perform real-time analytics on data. Scala's key advantages in helping developers achieve Fast Data include: