In this Lightbend podcast, we are joined by Jeremy Pollock, Senior Director of Product Management, to discuss the recent release of Akka Cloud Platform on AWS, what this means for Akka users, where Lightbend is heading with regards to cloud-based marketplace offerings and even our plans for hosted services. For more technical information, check out the Akka Cloud Platform deployment guide.
Note: this text has been edited for clarity and readability. Don't worry–you won't miss the transcribed stumbles, mumbles, and interviewer confusion ;-)
Oliver White: Welcome to this Lightbend podcast. This is Oliver White, Chief Storyteller and MC at Lightbend. And today, I'm sitting down with my colleague, Jeremy Pollock, Senior Director of Product Management, to discuss the recent release of Akka Cloud Platform on AWS, what this means for Akka users, where Lightbend is heading with regards to cloud-based marketplace offerings and even our plans for hosted services. All of this is powered by Akka and other open source projects from Lightbend. So I wanted to bring Jeremy on to our program today to discuss a little bit more. Jeremy, thanks for joining me today.
Jeremy Pollock: Thank you Oliver. Happy to be here. And happy to talk more about Akka and specifically Akka Cloud Platform.
OW: Excellent. So since this is your first podcast with Lightbend, you're probably not a household name yet. So why don't you tell us a little about yourself. Who are you? How do you describe what you do to your friends? And what are some of your top priorities for products at Lightbend?
JP: Cool. And if I could tell my friends what I do, I think I'd be a rich man by this point.
I think that most people have different views of product management, but for me, I've been in product management now for gosh, 20 years. Been in the tech industry, really focused on developers and developer platforms, been in the world of APIs and distributed Pub/Sub messaging environments, just really happy to always be thinking about how we can best or how I can best help teams build out solutions and problems for customers and for markets. And I'm a product person at heart, I love building things. And every so often I get a chance to hack around with code. And certainly, that's one of the things that I love about being here at Lightbend is just being able to work with a terrific platform like Akka.
We've been sketching out this future envisioning a world where the power of Akka–hugely scalable distributed state management and high speed application stream development–can be tapped into by engineers who aren't distributed system experts.
OW: Alright, thanks for that introduction. So let's get into it. We recently announced the availability of Akka Cloud Platform on the AWS marketplace. And one of the folks in the community asked me, "Hey, Oliver, hasn't Akka always kind of been designed for the cloud? What's the deal with the marketplace offering, what's different now?" And maybe we could just start there.
JP: Yeah, definitely. And great question. You know, I think Lightbend has been "cloud native" for well, gosh, before even cloud native was ever a thing, I suppose. And we've been in business for 10 years, helping customers build out mission-critical solutions based on this wonderful open source project called Akka. And what I think makes this announcement around Akka Cloud Platform and really even our product road map overall, what makes it most exciting is that Lightbend and Akka in many ways, is moving away from just a do-it-yourself platform where you're the expert, you're having to sit down and get into the guts of distributed computing and moving towards a future where developer productivity is radically transformed and in a positive way, of course. We all want to make the world better. Certainly, the past couple of years...
OW: By do-it-yourself platform, it kind of means that you've got this amazing Akka toolkit and you can kind of do anything with it, which means that you can do everything or [anything or] nothing.
JP: Yeah, exactly. And I think that's always one of the challenges with any technology like that, where the world is an open canvas and you can paint whatever picture you want to paint. But it's often to know where to start.
OW: Right.
JP: And it's great for the expert, but often not so great for the people who are trying to move a lot more quickly, which really is what all of us are trying to do today, move faster to solve problems in better ways. And with that in mind, and really kinda building upon our experiences in this world of distributed computing, Lightbend itself has been going through a transformation. We've been sketching out this future envisioning a world where the power of Akka–hugely scalable distributed state management and high speed application stream development–can be tapped into by engineers who aren't distributed system experts. And hopefully we'll get to some other podcasts as we move along here, where we talk about things like Akka Serverless which is in private beta right now.
Today I'll stick to Akka Cloud Platform. It really is this first public manifestation, a product of this vision of lowering the barriers to entry and adoption of Akka distributed technologies. And the audience might ask, "Why now? What's so important about this world, this moment in time, that we're all kind of experiencing?".
Is really being in the marketplace accessing Akka through AWS or accessing Akka through Google Cloud Platform, as an example, is that really that useful? And I think this world, this moment in time now that we live in, says it all. Sure, Lightbend and so many other enterprises out there today have been embarked upon digital transformation, cloud native journeys prior to COVID throwing us all a giant curveball. And there's no doubt that rapid roadmap acceleration was happening. I think lots of companies thought, we still had years to do all of this transformation and moving to the cloud and getting our services, our products up to the up to the level of meeting the expectations of the modern user out there, but all of that has been compressed.
Akka Cloud Platform is not about just rolling a Docker container and putting it into Kubernetes–it's about making it easier to do that operationally and also to tie into all of the great cloud services that you need to build out a full solution.
Companies are now scrambling to do this road map transformation in months as opposed to years. So removing any obstacles that we can find, making things easier, focusing on both the business and developer productivity, all this is that much more important today, and what has really driven this whole initiative, both the long-term vision, but also the short-term delivery of the Akka cloud platform. Ultimately that's what gets me excited about Akka cloud platform, it's the removal of obstacles, it could be procurement obstacles, which is great. Make it heck of a lot easier to go click on a button and buy Akka just like you buy shoes or books from your favorite e-commerce provider.
OW: Or face masks.
JP: Who would have thought that there would be a whole industry around face mask? We'll have to try to figure out an Akka angle there one day, but anyway, I digress. It's not just about the obstacles removed for the business folks, but it's also the developers, at the end of the day, we're all about developers, we love developers, we love empowering developers and engineers, making them that much more productive in building up their own distributed system solutions, but we want to allow them to build those solutions, address their business priorities, to be able to take advantage of the high-scale cloud systems out there today and give them a path, give them a better pattern to follow, to achieve their goals that much more quickly.
And while we've been making some headway and making it easier to integrate with cloud services like, "Hey, if you want to bring in your SQL database for your backing data store or use an Akka, use Amazon RDS, or if you want to have a Google Pub/Sub be the messaging backbone for your Akka powered application, do that, I think that's a great proof-point of the steps that we're taking in Akka, manifested now through Akka Cloud Platform, and just making it that much easier for development teams to adopt an architecture that really allows them to tap into the full power of the cloud. It's not about just rolling a Docker container and putting it into Kubernetes, it's about making easier to do that operationally, but also to tie into all of the great cloud services that you need to build out a full solution.
Now, I realize I've been talking a lot so far, and I'm talking as a PM, but what does it really mean? And so, first in the design patterns that we've delivered through Akka cloud platform, we're encouraging developers to use the approaches that we think allow them to take full advantage of raw Akka power, everything from Event Sourcing to CQRS, among other patterns, you can tap into this raw powers achieving, helping you achieve full compute utilization, database optimizations, include data back pressure support, without having to dig into the guts of Akka itself.
You still with Akka cloud platform need to understand Akka, you see Akka, you work with Akka but you don't always have to start with the guts of Akka itself in order to really achieve a whole lot. And second, with Akka Cloud Platform, with those templates of cloud service integration that I've talked about, these developers, the developers out there in the wild who are trying to build up solutions can manage or leverage those managed services. So it's not always about bringing your own database, it's about also tapping easily into a managed service for database capabilities or event buses or for classic messaging, but all of this allows more of a focus on the business requirements that developer is trying to address, to build up differentiating proprietary logic. As opposed to figuring out how all of the underlying technology needs to work.
OW: Yeah, that especially, you mentioned Covid a little while ago, and obviously how can we not acknowledge that or even refrain from thinking about it on an hourly basis, but the kind of on-ramp without having to deal with underlying tech or as you mentioned, procurement, all of that seems to be made even more challenging in a business environment where most people are working from home, so even if you had to go visit another floor of your office building in the past, a lot of the people that might be able to help you get started or rather move forward in the finance department. All of that would be within arm’s reach so to speak, and now we're all stuck in our little pods.
JP: Yeah, it sounds rather obvious now, for sure. I think before Covid, marketplaces were definitely gaining momentum. Companies were attracted to consolidated billing for sure, but I think definitely Covid has just brought about this world where we're all remote and we can't, to your point, walk down the hall and go harangue finance in order to get that provider approved in the budget. Having more control and be able to do that through the marketplace is just a complete efficiency boost if you will, when it comes to having teams get going and solving our problems much more quickly.
OW: Absolutely. So let's get into the meat of the discussion. We talked a little bit about why we created Akka Cloud Platform on AWS, let's talk about what it is, how it works, and who can use it and how?
JP: Definitely, cool. Well, as I said before, I love talking, especially love talking about my products. I think that's hopefully a good indicator of good product management skills, but let me get into it.
I'm especially excited about scaling up and down when it comes to the Akka Cluster Roles: when you're thinking about distributed state management and breaking up your microservices into really functional areas, having that ability to easily scale up read capacity to serve more query requirements against your APIs versus not necessarily scaling up the write side of your solution.
First Akka Cloud Platform, and I've talked a little bit about this or alluded to it, the Akka Cloud platform makes it this easier to deploy a managed applications and services built on Akka. The product comes with this Kubernetes operator, which takes all the necessary task that an SRE team, an ops team might have to take when standing up and running an Akka-based service and makes it available through an automatable tool. It makes it super simple to scale up, scale down, configure your logging, check on your Akka Cluster status. There are lots of things that the operator supports and those are some of them.
I'm especially excited about that scale up and scale down when it comes to the Akka Cluster Roles. So when you're thinking about distributed state management and breaking up your microservices into really functional areas, having that ability to easily scale up read capacity to serve more query requirements against your APIs versus not necessarily scaling up the write side of your solution. We're doing the opposite or both at the same time. It's super simple to really get a good handle on the operationalization of your Akka-based applications.
OW: So not to not to dig too deeply into this but scaling up and down by Cluster Role, this is something I believe we're supporting only in the Akka Cloud Platform for Amazon plug-in, is that correct?
JP: Correct. It'll also be there once we get to Google Cloud Platform, which is at the end of the month of March 2021. But yes, that is correct.
OW: So what this means is in a CQRS sort of pattern, you'll have a read and write side that are separate as described by the acronym itself. And in the case... [get confused a bit]… read sides scale up or down, write sides scale up or down some other roles independently of the entire cluster?
JP: Yes, and that ultimately gives you this awesome granular approach to managing costs, of course, but also performance and end user experience. If you again, are serving lots and lots of queries and your solution is struggling there, well, you could also use Auto Scaling at that point too to let it scale up for sure, but to have that fine-grained control to yield an optimal cost-benefit result, balancing costs and end user experience or solution experience at that very granular level.
OW: Gotcha, thanks for explaining that.
JP: Cool. Well, I think you explained a lot of it too, so thank you for that. But getting just to operations, past operations, which are of course super important in this world of ours. But it's not just about scaling up and scaling down, you need insight and you need to understand what's happening with your running system. So another big component of Akka Cloud Platform is our integrated telemetry. It gives developers and operations teams the ability to see into their distributed services.
No doubt, Akka is this great, super powerful platform, but at the end of the day, distributed computing and data are hard, hard to build and hard to manage. This is the very nature of the beast. When you get into production, what tools are you going to use to troubleshoot or deal with performance or errors or other instabilities that might arise. And yes, visualization and alerting tech, like, Datadog, Prometheus, Grafana, insert your favorite tool, they're all very important in this world of ongoing productions support.
The telemetry capabilities of the Akka Cloud Platform and Akka in general, really, are optimized for the sophisticated architecture of Akka and distributed computing itself. We have 10 years of expertise in this area, and in this open source platform. For example, you're going to need to know which parts of your event sourcing service or a CQRS-based solution is slowing down, that read or write side of the equation. Or if you're experiencing higher rates in a particular ops zone in your cloud environment, and you want to see during that time what segment of your distributed data, which users, which devices (in an IoT environment), might be impacted, Lightbend Telemetry gives you the granular system, distributed systems metrics to help answer these questions and troubleshoot and ultimately solve these operational problems.
OW: So let's take a moment to talk a little bit about telemetry. So I think there's two important things here. One is that if you're using Akka Cloud Platform, then telemetry is included. Currently, this is commercial software that we provide along with our subscription. So this is included in Akka Cloud Platfrom already, and I think that's good. The second part is reminding folks what Lightbend Telemetry is, how it does plug into tools like Grafana and Datadog, and how it provides something like a look into the black box of Akka. Could you just describe a little bit why it's not easy to monitor or get telemetry for Akka with existing tools out there that are not designed specifically for Akka?
JP: Yeah, we've spend lots and lots of time and we invest continuously in this in the proper instrumentation of the code itself, and ensuring that the functional paths, that the data flow paths are fully tracked or trackable within the context of the code base. And as code is executed, it's admitting the right metrics at the right point in time, so that you're, in essence, able to trace data, trace performance, trace errors at any point in time. And in a way that is readily attributable, if you will, for lack of a better phrase, to the logic, to the code itself, meaning that the metrics are a logging manifestation of the logic.
So you're not having to have this huge translation layer between what a metric is and how it relates to the logic that you've written and is running within production. So it's that mirror image of logic and capabilities that I think is most important. So again, that you're feeling like a hand slipping into a glove when it comes to looking at data, as opposed to always having to go from high level superficial metrics which might make sense to another system or another provider, but don't readily translate into the actual logic you've implemented within Akka itself.
OW: Thanks, that makes more sense. And I suppose that this is why we encourage developers to start using Lightbend Telemetry as they are coding. Could you tell us a little bit about why that makes a lot of sense?
JP: Sure. One, it's to get into the flow of actually leveraging telemetry in your development process to troubleshoot, understand where errors are happening, that makes sense, and it's fairly obvious. I think the second one, again, speaks to the always thinking about getting into production. And so one of the areas that we really focus on, that we have experience on, certainly is this whole notion of mission-critical, business-critical solutions.
If you think about production or how you're going to operate things at the very last moment in time, you're likely not to yield the best solution out there. You'll have to bolt in things, duct tape, try to nail things in and hope it all works. If you're thinking about telemetry and performance and error tracking throughout your development process and how you start to deploy code in staging and in testing environments and ultimately in the production, you're always thinking about a mission-critical approach to your problem as opposed to at the end, hoping for the best by slapping on a third-party solution potentially, or hoping that you've coded everything correctly and that Telemetry is going to give you the answer in the production environment.
So just having that experience, I think throughout the development process helps with the end result in production, but also helps with troubleshooting and developing for efficiency throughout that period.
OW: Yeah, we don't want our customers to be making headlines after Black Friday.
JP: That is definitely true.
OW: So let's talk a little bit about the integrations that Akka Cloud Platform offers for specifically AWS in this case.
JP: Yeah, definitely. And you know I think I'll tie it a little bit back to a comment that you made around Datadog, Prometheus, Grafana, and others, where Akka has this really an open integration architecture that's been true from day one. And no doubt that it is straightforward to integrate Akka into different environments is becoming that much more front and center in our product strategy and roadmap, where we want to not just build technologies and capabilities that make it easier to develop with Akka, but also make it easier to integrate Akka into other systems. Whether that's a managed streaming for Apache Kafka, for a Message Bus out there in AWS, or if you want to use Cassandra and leverage the DataStax Cassandra service, that's all part of your application.
The pay-as-you-go model with Akka Cloud Platform in the marketplace means that you pay for what you use. So it becomes a far cheaper way for smaller teams and projects that are just getting started out with an eye towards actually being in production with Akka at some point.
But also it speaks to the earlier point around being part of an ecosystem beyond just Akka itself, when it comes to observability, tracing, analytics, reporting, you know, all the various tools and technologies that you need to bring to bear to deliver your overall solution. So that's definitely becoming that much more critical to our path forward to simplifying integration across the board. We're part of an overall solution. And we believe that it's really important to have this great integration technology and approach to make it that much easier for developers to complete up that solution, specific to Akka Cloud Platform and these cloud services like MSK from AWS or the Cassandra service from DataStax, we have these templates.
We have these capabilities that allow a developer to go in and rapidly configure their Akka microservice to use DataStax or AWS as an example. And it sounds like a rather trivial problem to solve, but this one is all about velocity. It's all about just ticking off the things that a development team has to worry about. How are you going to stand up Kafka? How are you going to operate Kafka?
In this case, if you're using Kafka to shuttle data and events between microservices, for example, you know, rather than having to gain that expertise in Kafka, you can say, "Hey, I want to use a managed service, i.e. AWS managed streaming for Apache Kafka, and not have to worry about it. And that's important because operational expertise for all of these various components in a distributed system, one team can't have it all, many teams can't do it all. And so we want to make it easier to tap into these managed services that much more easily through our Akka Cloud Platform.
OW: Got it.
JP: The last piece of the puzzle I wanted to talk about, and I'll shift gears from helping just the developers to heck, making it a little bit easier for the business as well. Again, I've been about developers for a long time now, and so I love the whole notion of building and engineering, but I'm also a product manager who has to worry about business as well. But the pay-as-you-go model with Akka Cloud Platform in the marketplace, what that means is well, you pay for what you use. And then so it becomes a far cheaper way for smaller teams and projects that are just getting started out with an eye towards actually being in production with Akka, at some point, it becomes a much more accessible path to go.
Yes, we know Akka is open source, so you could always start there. Open source is free, but we know that there are many successful initiatives with Akka that are done and successful through a combination of open source and the commercial tech and Lightbend distributed system expertise that we've touched on throughout this talk today. So without a doubt, I feel like Akka Cloud Platform again, has lots of great technical goodness, but additionally through that pay as-you-go model, this makes it more business-accessible without having to jump full in on a more expensive Lightbend subscription.
Akka Cloud Platform is ideal for any teams looking to move fast. No sale cycle, no legal reviews, just click to subscribe and go, and you're well on your way towards having a battle hardened mission critical production platform to build out your solutions.
OW: Yeah, it sounds kind of ideal for ramping up a proof-of-concept with a small team for a new project and being able to show the bosses something running in production, all pretty much automated through marketplace.
JP: Yeah, and I think that the nuance there though, is that it's like a POC that could go to production. If you're hacking up a POC over a weekend and the code is horrible, then you might not want to go to production. But the great thing about Akka and showing Akka Cloud Platform is that you're acting as if you could go to production, that you are focused on solving mission-critical, business-critical problems. It may not be the best approach for a hack it over a weekend POC that you might want to throw away and never think about again, but I definitely think that there's that benefit of demonstrating focus and dedication to production quality, I think is a great path to go on with Akka Cloud Platform.
OW: Great. Let's talk about, while we get towards the end of our time today, who should be using the marketplace and how? So what's the difference between a Lightbend subscriber accessing this technology versus a Java developer who's hearing about Akka for the first time?
JP: Sure. First, if you're currently a Lightbend customer and you're already running on AWS or Google Cloud Platform, there's no need to take action. So don't worry, you're already on a great platform and using great technology.
If you're a Java or Scala developer and you're looking to get into a platform that gives you the path forward for successfully building these mission-critical elastic, highly scalable APIs and services, then Akka Cloud Platform form is a great way to do that, to get your hands not on the code, not just on the code, but also that production technology. The Kubernetes operator, the integrated telemetry, that'll get you well beyond trial mode, which is something that we just talked about just a second or two ago.
If you're already an Akka user, so you're not a customer, you're just using Akka out there in the world, you've been staying in the open source world during that time frame, but you want to start to get a taste of what it might like to run your production workloads using some of our commercial capabilities that previously were only available to you through an annual subscription, then just navigate to the AWS marketplace and click on subscribe. That pay-as-you-go purchasing model just is a great way to get some additional support and peace of mind and all of those tools to help you out in your own production workload management.
Ultimately though, I think just taking a step back, I think Akka Cloud Platform is ideal for any teams looking to move fast. No sale cycle, no legal reviews, just click to subscribe and go, and you're well on your way towards having a battle hardened mission critical production platform to build out your solutions.
OW: Yeah, that certainly sounds easy. So, as long as you have an AWS account, or access to that, you should be able to, as you described, just click subscribe and go.
JP: Correct, yeah.
OW: Before we end for the day, you mentioned Google Cloud Platform earlier, could you tell us what that's about, and what we're doing there?
JP: Great question. So, first, everything I've talked about today is as relevant for AWS, as it is for Google Cloud Platform, the former is available today, the latter is going to be available at the end of this month, March. Beyond those two cloud platforms, we're turning our attention to Azure and Red Hat, if there are other similar platforms that we should be running on, in manner like I've talked about today, that we've talked about today. Please don't hesitate to shoot me an email at PM@lightbend.com. Always happy to get that sort of input from our user base. That said, it's more than just getting into the cloud market places, we're doing more than just that, and more than just developer experience and tooling for existing Akka architectures. As I mentioned in the first part of our session here today, Lightbend, is at this beginning of a journey, of enabling developers of all types to tap into that distributed systems power of Akka.
And we're continuing this with a new managed service offering that we're working on today, it's called Akka Serverless, it's in private beta. I'm happy to get more users trying it out, and there's links available in our website, but with Akka Serverless, developers will be able to use whatever language they want beyond Java and Scala, it could be JavaScript, Python, Go among many others. And you can use this service to build out distributed state, or to include distributed state capabilities into their services, their APIs that they're building, without having to stand up any infrastructure, without having to manage any of that infrastructure. As I said, we're now in private beta, but we're planning to be out there in the public, mid-summer of this year, and definitely looking forward to again, having more and more developers tap into that power of Akka.
OW: Alright. Well, Jeremy Pollock, Senior Director of Product Management at Lightbend, and my dear colleague, it was very wonderful to speak with you today, thank you for joining us, and giving us such an excellent overview of what's going on, and I hope to welcome you back to the podcast program soon.
JP: Thank you for having me, it was a pleasure, and I look forward to more podcasts, and more discussions out there in the world when we come, when talking about Akka, and all that we're doing out there. So thank you.
