How To Distribute Application State with Akka Cluster - Part 3: Kubernetes & Monitoring

Proof Of Concept With Akka Cluster

In this series of blog posts, we walk you through a working Proof of Concept (PoC) built using Lightbend’s open source Scala programming language with the Akka distributed toolkit. In Part 1: Getting Started, we walked through building, testing, and running the PoC locally, with instrumentation and monitoring wired in from the very beginning using Lightbend Telemetry. In this Part 2: Docker and Local Deploy, we deploy to Docker to our local machine and test. In this Part 3: Kubernetes & Monitoring, we move our PoC to Kubernetes (on Minikube) and review some monitoring options. Finally, in Part 4: Source Code, we look at the Scala source code required to create the PoC.

In the previous post, we showed you how to test, run locally, package, deploy, and run our PoC in Docker with the associated repository, showed you how to load test via Gatling, and monitor via Lightbend Telemetry tooling and sandbox. In this part, we introduce Lightbend Console for Kubernetes (K8s), and then deploy our PoC in Minikube (desktop version of K8s) using YAML files provided in the repository. Again, we’ll load test with Gatling, but this time we’ll monitor our PoC in Lightbend Console. Finally, we’ll do a deep dive into the K8s YAML files being used for deployment.

Akka Complements K8s

Lightbend believes that Kubernetes is the best way to manage and orchestrate containers in the cloud; however, it doesn’t address the needs of programming and Reactive systems development. Akka on the other hand provides tools to solve the challenges associated with building distributed applications. When combined, Akka wonderfully complements Kubernetes by doing the heavy lifting of managing distributed state and communication, maintaining application consistency and delivery guarantees, while Kubernetes manages the underlying infrastructure.

Monitoring the PoC in K8s

As we move our focus to K8s we need to shift to new tools. One of those new tools is Lightbend Console, which enables you to observe and monitor Akka based applications running on K8s.

Setting up Lightbend Console

Running Lightbend Console in Minikube creates some additional requirements such as more memory and Helm, so it’s recommended that you start the Minikube set up here. Once you’ve completed these prerequisites and successfully installed Console you can move on to deploying the PoC.

Deploying the PoC in K8s

Now that we have Minikube up and running with Lightbend Console we’re ready to deploy our PoC.

The repository contains a subdirectory called K8s, which contains .yaml files for each of the three separate deployments required to successfully deploy and run our PoC on K8s. These include cassandra, endpoints, and nodes (a.k.a. cluster in previous installments). The directories looks like this:

K8s/
	cassandra/
		cassandra-db-deployment.yaml  
           		cassandra-db-service.yaml
	endpoints/
            	endpoint-deployment.yaml  
            	endpoint-role-binding.yaml  
            	endpoint-service-account.yaml  
            	endpoint-service.yaml
	nodes/
		akka-cluster-member-role.yaml  
           		node-deployment.yaml  
            	node-role-binding.yaml  
           		node-service-account.yaml  
           		node-service.yaml

Each of the directories contains yaml files for deployment and service. However, both endpoints and nodes require additional roles for proper operation and cluster formation.

To deploy the PoC in Minikube:

1. Open a terminal window and point your Docker environment at Minikube’s image registry by entering the following command:

     eval $(minikube docker-env)

2. From the PoC’s root directory build and push the PoC Docker image into the image registry with the following command:

     sbt docker:publishLocal

3. Move to the K8s subdirectory:

    cd K8s

4. Deploy Cassandra, Nodes, and Endpoints with the following commands:

    kubectl apply -f cassandra
    kubectl apply -f nodes
    kubectl apply -f endpoints
   

    

   If you don’t have a Lightbend Subscription then you’ll need to change the configuration files contained in the deployment yaml files for both nodes and endpoints respectively.

   On line 45 of nodes/node-deployment.yaml file you’ll need to use the file name nonsup-cluster-application-k8s.conf instead of cluster-application-k8s.conf.

   On line 26 of endpoints/endpoint-deployment.yaml file you’ll need to use the file name nonsup-endpoint-application-k8s.conf instead of endpoint-application-k8s.conf.

5. Verify proper operation by opening the Lightbend Console (Lightbend Subscription required)
   1. Open the Console by entering the following command:

        minikube service expose-es-console -n lightbend

   2. In the drop down, just below the top left,
      
      select the default namespace. You should now see the following.
      
6. Verify proper operation through the exposed endpoint.
   1. Find the Endpoint’s service URL with the following command:

        minikube service endpoint --url

   2. Issue the following command (remember to use the URL address retrieved from the previous step):

      curl -d '{"artifactId":1, "userId":"Michael"}' -H "Content-Type: application/json" -X POST /artifactState/setArtifactReadByUser

   3. Next verify the state has been persisted, with a query:

      curl '/artifactState/getAllStates?artifactId=1&userId=Michael'

      
      The response should look like this:

      {"artifactId":1,"artifactInUserFeed":false,"artifactRead":true,"userId":"Michael"}

Nodeport vs Ingress

In K8s, a NodePort exposes a service on each node’s IP at a static port (the NodePort). Whereas, an Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.

In a normal production environment, an Ingress is a more appropriate approach to exposing services outside of the cluster. However, this functionality isn’t currently provided out of the box, and therefore you need to install additional software to support an Ingress controller such as Ngnix, which is a common way to provide a load balancer to a K8s internal service. Many cloud providers also provide load balancers that can be added to K8s. It’s also worth noting here that some distributions, such as Openshift, provide built in load balancers through Routes.

In a local development environment running Minikube, a Nodeport is a simple and easy approach since no additional software is required and internal ports are mapped to the hosts external port. Normally in Minikube, a node is simply a virtual machine running on your laptop.

Load testing w/ Gatling

As we discussed in the two previous posts, Gatling is a highly capable load testing tool built on top of Akka that is used to load test HTTP endpoints.

In our previous example, we relied upon the PoC running in Docker, but now that we’re running the PoC in Minikube we’ll need to update our application.conf with the URL address of the endpoint that we found in Step 6 above. For example, if you comment out the localhost entry, un-comment the sample line for Minikube, and then update with the IP address found in the previous step, the application.conf should look something like this:

 
loadtest {
  # provides the base URL: http://localhost:8082
  # baseUrl = "http://localhost:8082"
  # sample baseURL when running locally in docker
  #baseUrl = "http://172.18.0.5:8082"
  # sample baseURL when running locally in minikube
  baseUrl = "http://192.168.39.237:30082"
}

To start Gatling load test:

In a new terminal switch to the gatling directory from the PoCs root directory.
Enter the command:

 
  sbt gatling:test

While the load test is running we recommend jumping to the next section and taking a look at the metrics being collected real time by Lightbend Telemetry in Lightbend Console.

Once the test has completed, a message is displayed referring you to the specific report. You can find all reports created by Gatling, which are stored the ./gatling/target/gatling directory that is created each time a load test is run.

Experiment and Scale

If you take a look at the file gatling/src/test/scala/com/lightbend/gatling/ArtifactStateScenario you’ll see the default “set up” near the bottom:

 
  scn.inject(rampUsers(1000) during (5 minutes))

This set up scenario ramps up to 1000 users over a period of five minutes. Normally this test should run without any errors. You can easily modify either the number of users, ramp up time, or both and run again to see how things work.

Another experient you can try is scaling up the number of node replicas up and down. For example from the K8s directory enter the following command to scale replicas up from three to five:

 
  kubectl scale --replicas=5 -f nodes/node-deployment.yaml

To scale nodes back down, simply reuse the command(s) above and change the replica count to the original values of three.

You can also try scaling up the number of endpoint replicas too. For example, from the K8s directory enter the following command to scale replicats up from one to two:

 
  kubectl scale --replicas=2 -f endpoints/endpoint-deployment.yaml

Ultimately, you’ll probably find that Cassandra becomes your bottleneck since it’s only running in a single pod.

A number of other “set ups” have been provided in ArtifactStateScenario for your experimentation, but they have been commented out. There is one particular “set up” that comes directly from the Gatling documentation here, which should overload the PoC and cause a number of timeouts. Look through the various metrics and try to determine where the problems originate.

Browsing through the Metrics captured by Lightbend Telemetry

You can find the Grafana home page that’s provided by Lightbend Console by clicking on the Grafana icon in the top left corner of Console

We recommended that you take a look at the following dashboards while Gatling load testing is running (previous section).

• Akka Actors
• Akka and Lagom Persistence
• Akka HTTP and Play Framework Servers
• Akka Cluster
• Akka Cluster Sharding
• Akka Dispatchers
• JVM Metrics

Deep Dive Into K8s YAML Files

The repository contains a subdirectory called K8s, which contains YAML files for each of the three separate deployments required to successfully deploy and run our PoC on K8s. These include Cassandra, Nodes, and Endpoints.

Each of the directories contains YAML files for deployment and service. However, both endpoints and nodes require additional roles for proper operation and cluster formation.

Cassandra

We’re not going to spend a lot of time here on Cassandra, as we simply used the Kubernetes Kompose utility to convert our docker-compose-cassandra.yml file into K8s YAML files. It’s usually not recommended to run production databases inside of K8s unless they’re managed by an operator. However, for our PoC, these YAML files work well enough to bring up a single Cassandra pod in K8s.

Nodes

We’re starting with nodes here because the akka-cluster-member-role.yaml is used by both nodes and endpoints:

akka-cluster-member-role.yaml

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: akka-cluster-member
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
 

This YAML file is used to create the akka-cluster-member role that’s required by Akka Management, which uses the K8s API to find the other pods (nodes) for proper cluster formation.

node-deployment.yaml

We will highlight individual pieces of this file, since there’s so much going on here. This deployment file starts with traditional metadata settings:

 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: node
spec:
  replicas: 3
  selector:
     matchLabels:
       app: ArtifactStateCluster
  template:
    metadata:
      labels:
        app: ArtifactStateCluster
        tag: clusternode
      annotations:
        prometheus.io/scrape: 'true'
 

Note above:

1. We start with three replicas or pods upon initialization.
2. We’re using the app: ArtifactStateCluster as a selector for Akka Management to find the pods during the cluster bootstrap process.
3. The annotation for prometheus.io/scrape: 'true' is used to tell the Lightbend Console to scrape metrics generated by Lightbend Telemetry.

Next comes the spec section of the deployment:

 
    spec:
      serviceAccountName: nodes-sa
      containers:
      - name: node
        image: akka-typed-blog-distributed-state/cluster:0.1.0
 

Note above:

1. The serviceAccountName: nodes-sa is created by node-service-account.yaml, and used to identify role bindings defined by node-role-bindings.yaml.
2. The Docker image is identified by image: akka-typed-blog-distributed-state/cluster:0.1.0.

Next, comes the K8s health prob configurations:

        #health
        readinessProbe:
          httpGet:
            path: /ready
            port: akka-mgmt-http
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /alive
            port: akka-mgmt-http
          initialDelaySeconds: 90
          periodSeconds: 30
        #health

Note above: Akka Management, which we’ve included in our build, provides built-in implementations for K8s health readiness and liveness probes. You simply need to provide the proper configuration in your deployment YAML file to take advantage of these features. Since we’re using Akka Cluster, we need to give Akka enough time to find all nodes and then form the cluster before accepting outside requests.

Next, we’ll be taking a look at environment variables passed from the deployment:

        env:
        - name: HOSTNAME
          valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.podIP
        - name: CASSANDRA_CONTACT_POINT1
          value: "cassandra-db"
        - name: JAVA_OPTS
          value: "-Dconfig.resource=cluster-application-k8s.conf"

Note above:

1. HOSTNAME is exposing the pods IP address.
2. CASSANDRA_CONTACT_POINT1 is exposing the DNS name of the Cassandra database.
3. JAVA_OPTS is used to configure the desired configuration file.

Environment variables are accessed through typesafe config files using the notation of {?ENV_NAME}.

Next, comes port naming:

        ports:
        # akka remoting
        - name: remoting
          containerPort: 2552
          protocol: TCP
        # external http
        - name: akka-mgmt-http
          containerPort: 8558
          protocol: TCP
        - name: node-metrics
          containerPort: 9001
      restartPolicy: Always
 

Note above:

1. remoting is used by Akka Cluster gossip.
2. akka-mgmt-http is used by Akka Management.
3. node-metrics is the port used by Console / Prometheus to collect Telemetry metrics.

node-role-binding.yaml

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nodes-akka-cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: akka-cluster-member
subjects:
- kind: ServiceAccount
  namespace: default
  name: nodes-sa

node-role-binding.yaml binds the akka-cluster-member role with the service account nodes-sa.

node-service.yaml

 
apiVersion: v1
kind: Service
metadata:
  name: node
spec:
  type: NodePort
  ports:
    - name: akka-mgmt-http
      protocol: TCP
      port: 8558
      targetPort: akka-mgmt-http
      nodePort: 30558
  selector:
    tag: clusternode
 

node-service.yaml exposes an external Nodeport on port 30558 on each running node. In our case this is the single VM hosting Minikube. The incoming traffic is forwarded to pods identified by the selector “clusternode” in a round-robin fashion.

Port 8558 is being used for Akka Management’s HTTP interface. To find out more please see the documentation here.

For example, if you wanted to see what Akka views as cluster membership you could enter the following in a terminal to find the IP:

$ minikube service node --url
http://192.168.39.73:30558
 

Then, using the response above you could use the following to find Akka’s cluster membership:

 
$ curl http://192.168.39.73:30558/cluster/members | python -m json.tool
 

The result might look like this:

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1012  100  1012    0     0  43491      0 --:--:-- --:--:-- --:--:-- 44000
{
    "leader": "akka://ArtifactStateCluster@172.17.0.11:2552",
    "members": [
        {
            "node": "akka://ArtifactStateCluster@172.17.0.11:2552",
            "nodeUid": "5081129479427795119",
            "roles": [
                "sharded",
                "k8s",
                "dc-default"
            ],
            "status": "Up"
        },
        {
            "node": "akka://ArtifactStateCluster@172.17.0.13:2552",
            "nodeUid": "-869599694544277744",
            "roles": [
                "sharded",
                "k8s",
                "dc-default"
            ],
            "status": "Up"
        },
        {
            "node": "akka://ArtifactStateCluster@172.17.0.14:2552",
            "nodeUid": "-9132943336702189323",
            "roles": [
                "endpoint",
                "k8s",
                "dc-default"
            ],
            "status": "Up"
        },
        {
            "node": "akka://ArtifactStateCluster@172.17.0.15:2552",
            "nodeUid": "4657711064622987928",
            "roles": [
                "sharded",
                "k8s",
                "dc-default"
            ],
            "status": "Up"
        }
    ],
    "oldest": "akka://ArtifactStateCluster@172.17.0.11:2552",
    "oldestPerRole": {
        "dc-default": "akka://ArtifactStateCluster@172.17.0.11:2552",
        "endpoint": "akka://ArtifactStateCluster@172.17.0.14:2552",
        "k8s": "akka://ArtifactStateCluster@172.17.0.11:2552",
        "sharded": "akka://ArtifactStateCluster@172.17.0.11:2552"
    },
    "selfNode": "akka://ArtifactStateCluster@172.17.0.13:2552",
    "unreachable": []
}

If you were to repeat this command over and over you’ll notice the “selfNode” changes in round-robin fashion through each of the members.

node-service-account.yaml

 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nodes-sa
 

node-service-account.yaml simply creates the service account “nodes-sa” used for bindings between roles and the deployment.

That pretty much covers it for our nodes YAML files, next we’ll take a look at our endpoints YAML files.

Endpoints

The endpoints YAML files follow the same basic pattern as the nodes, not to mention that both deployments use the exact same Docker image, so we’re just going to highlight some of the differences in this section.

First, let's take a look at role bindings, as we’re re-using the akka-cluster-member role.

endpoint-role-binding.yaml

 
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: endpoint-akka-cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: akka-cluster-member
subjects:
- kind: ServiceAccount
  namespace: default
  name: endpoints-sa
 

We’re still using a unique service account for endpoints, but because Akka Management uses the K8s API during the cluster bootstrapping process to find other pods, we’re sharing the same akka-cluster-member for both nodes and endpoints.

Another difference with endpoints is that we’re exposing two Nodeports in our service definition.

endpoint-service.yaml

 
apiVersion: v1
kind: Service
metadata:
  name: endpoint
spec:
  type: NodePort
  ports:
    - name: "8082"
      protocol: TCP
      port: 8082
      targetPort: 8082
      nodePort: 30082
    - name: akka-mgmt-http
      protocol: TCP
      port: 8558
      targetPort: akka-mgmt-http
      nodePort: 30559
  selector:
    tag: endpoint
 

In the case of endpoints, we’re exposing ports 8082 and 8558 on the NodePorts 30082 and 30559 respectively. Port 30082 is serving up our PoC’s API, while Port 30559 provides access to Akka Management. We’re already using port 30558 for the akka-mgmt-http with nodes, so we had to choose a different port. It’s also possible to leave the nodePort: setting off, and let K8s assign an open available port.

You can use the following Minikube command to verify the IP and ports of the endpoints with the following command in a terminal:

 
$ minikube service endpoint --url
http://192.168.39.73:30082
http://192.168.39.73:30559
 

As you can see, the IP and both ports are being reported.

Shutting Down

Undeploying the PoC

To optionally undeploy the PoC use these commands in a terminal from the K8s directory:

 
  kubectl delete -f endpoints
  kubectl delete -f nodes
  kubectl delete -f cassandra
 

Uninstalling Console

You can find instructions on uninstalling Lightbend Console here.

Shutting down Minikube

To shut down Minikube enter the following in a terminal:

minikube stop

Optionally Delete Minikube

minikube delete

Conclusion

In this blog post, we looked at how to use sbt’s Native Packager to build and deploy our PoC to Minikube / Kubernetes. Again, we’ve also shown you how to run load testing with Gatling, experiment with scaling, and how to view the instrumentation for the PoC using Lightbend Console. Finally, we did a deep dive into the K8s deployment YAML files we’re using for the PoC.

Next up in this series is Part 4: Source Code, in which we look at the Scala source code, including Akka’s 2.6.x functional Typed Persistent Actors, required to create the PoC. Or, if you'd like to set up a demo or speak to someone about using Akka Cluster in your organization, click below to get started in that direction:

SEE AKKA CLUSTER IN ACTION

Share

Author(s)

Michael Read

Senior Technical Consultant, Lightbend, Inc.

| github: michael-read

Michael Read is an experienced, business-focused architect and systems engineer with extensive experience in Reactive architectures (optimizing for resilience, scalability, and responsiveness). He has a deep understanding of cloud-native architecture, application deployment, container orchestration, and resilience and fault tolerance of reactive systems. He’s implemented Reactive systems with modern technologies–including Akka, Akka Streams, Play, Lagom, Elasticsearch, Slick, Scala, Spark, Kafka, Kubernetes (K8s), and Cassandra–and has held Chief Architect and CTO roles prior to coming to Lightbend as well as mentored engineering teams at companies that include American Express.

Comments

View All Posts or Filter By Tag