Proof Of Concept With Akka Cluster
In this series of blog posts, we walk you through a working Proof of Concept (PoC) built using Lightbend’s open source Scala programming language with the Akka distributed toolkit. In Part 1: Getting Started, we walked through building, testing, and running the PoC locally, with instrumentation and monitoring wired in from the very beginning using Lightbend Telemetry. In this Part 2: Docker and Local Deploy, we deploy to Docker to our local machine and test. In this Part 3: Kubernetes & Monitoring, we move our PoC to Kubernetes (on Minikube) and review some monitoring options. Finally, in Part 4: Source Code, we look at the Scala source code required to create the PoC.
In the previous post, we showed you how to test, run locally, package, deploy, and run our PoC in Docker with the associated repository, showed you how to load test via Gatling, and monitor via Lightbend Telemetry tooling and sandbox. In this part, we introduce Lightbend Console for Kubernetes (K8s), and then deploy our PoC in Minikube (desktop version of K8s) using YAML files provided in the repository. Again, we’ll load test with Gatling, but this time we’ll monitor our PoC in Lightbend Console. Finally, we’ll do a deep dive into the K8s YAML files being used for deployment.
Akka Complements K8s
Lightbend believes that Kubernetes is the best way to manage and orchestrate containers in the cloud; however, it doesn’t address the needs of programming and Reactive systems development. Akka on the other hand provides tools to solve the challenges associated with building distributed applications. When combined, Akka wonderfully complements Kubernetes by doing the heavy lifting of managing distributed state and communication, maintaining application consistency and delivery guarantees, while Kubernetes manages the underlying infrastructure.
Monitoring the PoC in K8s
As we move our focus to K8s we need to shift to new tools. One of those new tools is Lightbend Console, which enables you to observe and monitor Akka based applications running on K8s.
Setting up Lightbend Console
Running Lightbend Console in Minikube creates some additional requirements such as more memory and Helm, so it’s recommended that you start the Minikube set up here. Once you’ve completed these prerequisites and successfully installed Console you can move on to deploying the PoC.
If you don’t have a Lightbend Subscription then you’ll need to skip this step.
Deploying the PoC in K8s
Now that we have Minikube up and running with Lightbend Console we’re ready to deploy our PoC.
The repository contains a subdirectory called K8s, which contains .yaml files for each of the three separate deployments required to successfully deploy and run our PoC on K8s. These include cassandra, endpoints, and nodes (a.k.a. cluster in previous installments). The directories looks like this:
K8s/
cassandra/
cassandra-db-deployment.yaml
cassandra-db-service.yaml
endpoints/
endpoint-deployment.yaml
endpoint-role-binding.yaml
endpoint-service-account.yaml
endpoint-service.yaml
nodes/
akka-cluster-member-role.yaml
node-deployment.yaml
node-role-binding.yaml
node-service-account.yaml
node-service.yaml
Each of the directories contains yaml files for deployment and service. However, both endpoints and nodes require additional roles for proper operation and cluster formation.
To deploy the PoC in Minikube:
- Open a terminal window and point your Docker environment at Minikube’s image registry by entering the following command:
eval $(minikube docker-env)
- From the PoC’s root directory build and push the PoC Docker image into the image registry with the following command:
sbt docker:publishLocal
- Move to the K8s subdirectory:
cd K8s
- Deploy Cassandra, Nodes, and Endpoints with the following commands:
kubectl apply -f cassandra kubectl apply -f nodes kubectl apply -f endpoints
If you don’t have a Lightbend Subscription then you’ll need to change the configuration files contained in the deployment yaml files for both nodes and endpoints respectively.
On line 45 of nodes/node-deployment.yaml file you’ll need to use the file name nonsup-cluster-application-k8s.conf instead of cluster-application-k8s.conf.
On line 26 of endpoints/endpoint-deployment.yaml file you’ll need to use the file name nonsup-endpoint-application-k8s.conf instead of endpoint-application-k8s.conf.
- Verify proper operation by opening the Lightbend Console (Lightbend Subscription required)
- Open the Console by entering the following command:
minikube service expose-es-console -n lightbend
- In the drop down, just below the top left,
select the default namespace. You should now see the following.
- Open the Console by entering the following command:
- Verify proper operation through the exposed endpoint.
- Find the Endpoint’s service URL with the following command:
minikube service endpoint --url
- Issue the following command (remember to use the URL address retrieved from the previous step):
curl -d '{"artifactId":1, "userId":"Michael"}' -H "Content-Type: application/json" -X POST /artifactState/setArtifactReadByUser
- Next verify the state has been persisted, with a query:
curl '/artifactState/getAllStates?artifactId=1&userId=Michael'
The response should look like this:{"artifactId":1,"artifactInUserFeed":false,"artifactRead":true,"userId":"Michael"}
- Find the Endpoint’s service URL with the following command:
Nodeport vs Ingress
In K8s, a NodePort exposes a service on each node’s IP at a static port (the NodePort). Whereas, an Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.
In a normal production environment, an Ingress is a more appropriate approach to exposing services outside of the cluster. However, this functionality isn’t currently provided out of the box, and therefore you need to install additional software to support an Ingress controller such as Ngnix, which is a common way to provide a load balancer to a K8s internal service. Many cloud providers also provide load balancers that can be added to K8s. It’s also worth noting here that some distributions, such as Openshift, provide built in load balancers through Routes.
In a local development environment running Minikube, a Nodeport is a simple and easy approach since no additional software is required and internal ports are mapped to the hosts external port. Normally in Minikube, a node is simply a virtual machine running on your laptop.
Load testing w/ Gatling
As we discussed in the two previous posts, Gatling is a highly capable load testing tool built on top of Akka that is used to load test HTTP endpoints.
In our previous example, we relied upon the PoC running in Docker, but now that we’re running the PoC in Minikube we’ll need to update our application.conf with the URL address of the endpoint that we found in Step 6 above. For example, if you comment out the localhost entry, un-comment the sample line for Minikube, and then update with the IP address found in the previous step, the application.conf should look something like this:
loadtest {
# provides the base URL: http://localhost:8082
# baseUrl = "http://localhost:8082"
# sample baseURL when running locally in docker
#baseUrl = "http://172.18.0.5:8082"
# sample baseURL when running locally in minikube
baseUrl = "http://192.168.39.237:30082"
}
To start Gatling load test:
In a new terminal switch to the gatling directory from the PoCs root directory.
Enter the command:
sbt gatling:test
While the load test is running we recommend jumping to the next section and taking a look at the metrics being collected real time by Lightbend Telemetry in Lightbend Console.
Once the test has completed, a message is displayed referring you to the specific report. You can find all reports created by Gatling, which are stored the ./gatling/target/gatling directory that is created each time a load test is run.
Experiment and Scale
If you take a look at the file gatling/src/test/scala/com/lightbend/gatling/ArtifactStateScenario you’ll see the default “set up” near the bottom:
scn.inject(rampUsers(1000) during (5 minutes))
This set up scenario ramps up to 1000 users over a period of five minutes. Normally this test should run without any errors. You can easily modify either the number of users, ramp up time, or both and run again to see how things work.
Another experient you can try is scaling up the number of node replicas up and down. For example from the K8s directory enter the following command to scale replicas up from three to five:
kubectl scale --replicas=5 -f nodes/node-deployment.yaml
To scale nodes back down, simply reuse the command(s) above and change the replica count to the original values of three.
You can also try scaling up the number of endpoint replicas too. For example, from the K8s directory enter the following command to scale replicats up from one to two:
kubectl scale --replicas=2 -f endpoints/endpoint-deployment.yaml
Note: to properly scale endpoints you should really be using a properly configured load balancer through an Ingress to route traffic instead of the NodePort we’re currently using. Openshift / Minishift provides a built-in route that supports load balancing.
Ultimately, you’ll probably find that Cassandra becomes your bottleneck since it’s only running in a single pod.
A number of other “set ups” have been provided in ArtifactStateScenario for your experimentation, but they have been commented out. There is one particular “set up” that comes directly from the Gatling documentation here, which should overload the PoC and cause a number of timeouts. Look through the various metrics and try to determine where the problems originate.
Browsing through the Metrics captured by Lightbend Telemetry
You can find the Grafana home page that’s provided by Lightbend Console by clicking on the Grafana icon in the top left corner of Console
We recommended that you take a look at the following dashboards while Gatling load testing is running (previous section).
- Akka Actors
- Akka and Lagom Persistence
- Akka HTTP and Play Framework Servers
- Akka Cluster
- Akka Cluster Sharding
- Akka Dispatchers
- JVM Metrics
Deep Dive Into K8s YAML Files
The repository contains a subdirectory called K8s, which contains YAML files for each of the three separate deployments required to successfully deploy and run our PoC on K8s. These include Cassandra, Nodes, and Endpoints.
Each of the directories contains YAML files for deployment and service. However, both endpoints and nodes require additional roles for proper operation and cluster formation.
Cassandra
We’re not going to spend a lot of time here on Cassandra, as we simply used the Kubernetes Kompose utility to convert our docker-compose-cassandra.yml file into K8s YAML files. It’s usually not recommended to run production databases inside of K8s unless they’re managed by an operator. However, for our PoC, these YAML files work well enough to bring up a single Cassandra pod in K8s.
Nodes
We’re starting with nodes here because the akka-cluster-member-role.yaml is used by both nodes and endpoints:
akka-cluster-member-role.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: akka-cluster-member
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["get", "watch", "list"]
This YAML file is used to create the akka-cluster-member role that’s required by Akka Management, which uses the K8s API to find the other pods (nodes) for proper cluster formation.
node-deployment.yaml
We will highlight individual pieces of this file, since there’s so much going on here. This deployment file starts with traditional metadata settings:
apiVersion: apps/v1
kind: Deployment
metadata:
name: node
spec:
replicas: 3
selector:
matchLabels:
app: ArtifactStateCluster
template:
metadata:
labels:
app: ArtifactStateCluster
tag: clusternode
annotations:
prometheus.io/scrape: 'true'
Note above:
- We start with three replicas or pods upon initialization.
- We’re using the app: ArtifactStateCluster as a selector for Akka Management to find the pods during the cluster bootstrap process.
- The annotation for prometheus.io/scrape: 'true' is used to tell the Lightbend Console to scrape metrics generated by Lightbend Telemetry.
Next comes the spec section of the deployment:
spec:
serviceAccountName: nodes-sa
containers:
- name: node
image: akka-typed-blog-distributed-state/cluster:0.1.0
Note above:
- The serviceAccountName: nodes-sa is created by node-service-account.yaml, and used to identify role bindings defined by node-role-bindings.yaml.
- The Docker image is identified by image: akka-typed-blog-distributed-state/cluster:0.1.0.
Next, comes the K8s health prob configurations:
#health
readinessProbe:
httpGet:
path: /ready
port: akka-mgmt-http
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
httpGet:
path: /alive
port: akka-mgmt-http
initialDelaySeconds: 90
periodSeconds: 30
#health
Note above: Akka Management, which we’ve included in our build, provides built-in implementations for K8s health readiness and liveness probes. You simply need to provide the proper configuration in your deployment YAML file to take advantage of these features. Since we’re using Akka Cluster, we need to give Akka enough time to find all nodes and then form the cluster before accepting outside requests.
Next, we’ll be taking a look at environment variables passed from the deployment:
env:
- name: HOSTNAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: CASSANDRA_CONTACT_POINT1
value: "cassandra-db"
- name: JAVA_OPTS
value: "-Dconfig.resource=cluster-application-k8s.conf"
Note above:
- HOSTNAME is exposing the pods IP address.
- CASSANDRA_CONTACT_POINT1 is exposing the DNS name of the Cassandra database.
- JAVA_OPTS is used to configure the desired configuration file.
Environment variables are accessed through typesafe config files using the notation of {?ENV_NAME}.
Next, comes port naming:
ports:
# akka remoting
- name: remoting
containerPort: 2552
protocol: TCP
# external http
- name: akka-mgmt-http
containerPort: 8558
protocol: TCP
- name: node-metrics
containerPort: 9001
restartPolicy: Always
Note above:
- remoting is used by Akka Cluster gossip.
- akka-mgmt-http is used by Akka Management.
- node-metrics is the port used by Console / Prometheus to collect Telemetry metrics.
node-role-binding.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nodes-akka-cluster
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: akka-cluster-member
subjects:
- kind: ServiceAccount
namespace: default
name: nodes-sa
node-role-binding.yaml binds the akka-cluster-member role with the service account nodes-sa.
node-service.yaml
apiVersion: v1
kind: Service
metadata:
name: node
spec:
type: NodePort
ports:
- name: akka-mgmt-http
protocol: TCP
port: 8558
targetPort: akka-mgmt-http
nodePort: 30558
selector:
tag: clusternode
node-service.yaml exposes an external Nodeport on port 30558 on each running node. In our case this is the single VM hosting Minikube. The incoming traffic is forwarded to pods identified by the selector “clusternode” in a round-robin fashion.
Port 8558 is being used for Akka Management’s HTTP interface. To find out more please see the documentation here.
For example, if you wanted to see what Akka views as cluster membership you could enter the following in a terminal to find the IP:
$ minikube service node --url
http://192.168.39.73:30558
Then, using the response above you could use the following to find Akka’s cluster membership:
$ curl http://192.168.39.73:30558/cluster/members | python -m json.tool
The result might look like this:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1012 100 1012 0 0 43491 0 --:--:-- --:--:-- --:--:-- 44000
{
"leader": "akka://ArtifactStateCluster@172.17.0.11:2552",
"members": [
{
"node": "akka://ArtifactStateCluster@172.17.0.11:2552",
"nodeUid": "5081129479427795119",
"roles": [
"sharded",
"k8s",
"dc-default"
],
"status": "Up"
},
{
"node": "akka://ArtifactStateCluster@172.17.0.13:2552",
"nodeUid": "-869599694544277744",
"roles": [
"sharded",
"k8s",
"dc-default"
],
"status": "Up"
},
{
"node": "akka://ArtifactStateCluster@172.17.0.14:2552",
"nodeUid": "-9132943336702189323",
"roles": [
"endpoint",
"k8s",
"dc-default"
],
"status": "Up"
},
{
"node": "akka://ArtifactStateCluster@172.17.0.15:2552",
"nodeUid": "4657711064622987928",
"roles": [
"sharded",
"k8s",
"dc-default"
],
"status": "Up"
}
],
"oldest": "akka://ArtifactStateCluster@172.17.0.11:2552",
"oldestPerRole": {
"dc-default": "akka://ArtifactStateCluster@172.17.0.11:2552",
"endpoint": "akka://ArtifactStateCluster@172.17.0.14:2552",
"k8s": "akka://ArtifactStateCluster@172.17.0.11:2552",
"sharded": "akka://ArtifactStateCluster@172.17.0.11:2552"
},
"selfNode": "akka://ArtifactStateCluster@172.17.0.13:2552",
"unreachable": []
}
If you were to repeat this command over and over you’ll notice the “selfNode” changes in round-robin fashion through each of the members.
node-service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nodes-sa
node-service-account.yaml simply creates the service account “nodes-sa” used for bindings between roles and the deployment.
That pretty much covers it for our nodes YAML files, next we’ll take a look at our endpoints YAML files.
Endpoints
The endpoints YAML files follow the same basic pattern as the nodes, not to mention that both deployments use the exact same Docker image, so we’re just going to highlight some of the differences in this section.
First, let's take a look at role bindings, as we’re re-using the akka-cluster-member role.
endpoint-role-binding.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: endpoint-akka-cluster
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: akka-cluster-member
subjects:
- kind: ServiceAccount
namespace: default
name: endpoints-sa
We’re still using a unique service account for endpoints, but because Akka Management uses the K8s API during the cluster bootstrapping process to find other pods, we’re sharing the same akka-cluster-member for both nodes and endpoints.
Another difference with endpoints is that we’re exposing two Nodeports in our service definition.
endpoint-service.yaml
apiVersion: v1
kind: Service
metadata:
name: endpoint
spec:
type: NodePort
ports:
- name: "8082"
protocol: TCP
port: 8082
targetPort: 8082
nodePort: 30082
- name: akka-mgmt-http
protocol: TCP
port: 8558
targetPort: akka-mgmt-http
nodePort: 30559
selector:
tag: endpoint
In the case of endpoints, we’re exposing ports 8082 and 8558 on the NodePorts 30082 and 30559 respectively. Port 30082 is serving up our PoC’s API, while Port 30559 provides access to Akka Management. We’re already using port 30558 for the akka-mgmt-http with nodes, so we had to choose a different port. It’s also possible to leave the nodePort: setting off, and let K8s assign an open available port.
You can use the following Minikube command to verify the IP and ports of the endpoints with the following command in a terminal:
$ minikube service endpoint --url
http://192.168.39.73:30082
http://192.168.39.73:30559
As you can see, the IP and both ports are being reported.
Shutting Down
Undeploying the PoC
To optionally undeploy the PoC use these commands in a terminal from the K8s directory:
kubectl delete -f endpoints
kubectl delete -f nodes
kubectl delete -f cassandra
Uninstalling Console
You can find instructions on uninstalling Lightbend Console here.
Shutting down Minikube
To shut down Minikube enter the following in a terminal:
minikube stop
Note: you can simply stop Minikube without stopping or undeploying the PoC.
Optionally Delete Minikube
minikube delete
Conclusion
In this blog post, we looked at how to use sbt’s Native Packager to build and deploy our PoC to Minikube / Kubernetes. Again, we’ve also shown you how to run load testing with Gatling, experiment with scaling, and how to view the instrumentation for the PoC using Lightbend Console. Finally, we did a deep dive into the K8s deployment YAML files we’re using for the PoC.
Next up in this series is Part 4: Source Code, in which we look at the Scala source code, including Akka’s 2.6.x functional Typed Persistent Actors, required to create the PoC. Or, if you'd like to set up a demo or speak to someone about using Akka Cluster in your organization, click below to get started in that direction: