Multi-cluster setup

Overview of an example setup

• Keptn Control plane

  • The control plane is the minimum set of components, which are required to run a Keptn and to manage projects, stages, and services, to handle events, and to provide integration points.
  • The control plane orchestrates the task sequences defined in Shipyard, but does not actively execute the tasks.
  • Minimum Cluster size

• Keptn Execution plane

  • The execution plane consists of all Keptn-services that are required to process all tasks (like deployment, test, etc.).
  • The execution plane is the cluster where you deploy your application too and execute certain tasks of a task sequence.
  • Minimum Cluster size

Create or bring two (or more) Kubernetes clusters

To operate Keptn in a multi-cluster setup, you need obviously at least two Kubernetes clusters:

1. One that runs Keptn as control plane
2. The second one that runs the execution-plane services for deploying, testing, executing remediation actions, etc.

To create a Kubernetes cluster, please follow the instructions here.

Prerequisites

• keptn CLI
• helm CLI

Install Keptn Control plane

The Control Plane of Keptn is responsible for orchestrating your processes for continuous delivery or automated operations.

• Before starting the installation, make yourself familiar with the ways of exposing Keptn as explained here. Then come back and continue installing Keptn control plane.

• To install the control plane, execute keptn install with the option you chose for exposing Keptn:

  keptn install --endpoint-service-type=[LoadBalancer, NodePort, ClusterIP]
  

• Before continuing, please retrieve:

  • KEPTN_ENDPOINT: Follow the instructions here, depending on the option you chose for exposing Keptn.
  • KEPTN_API_TOKEN: Follow the instructions here.

Install Keptn Execution plane

In this release of Keptn, the execution plane services for deployment (helm-service) and testing (jmeter-service) can be installed via Helm Charts.

Please find the Helm Charts here:

• helm-service: GitHub Release for 0.11.4 at Assets > helm-service-0.11.4.tgz

• jmeter-service: GitHub Release for 0.11.4 at Assets > jmeter-service-0.11.4.tgz

How to deploy an execution plane services?

• Download the values.yaml from the release branch, e.g., for the jmeter-service:

  wget https://raw.githubusercontent.com/keptn/keptn/release-0.11.4/jmeter-service/chart/values.yaml
  

• Edit the values.yaml to connect the services to the Keptn control plane, identified by its endpoint and API token. Therefore, set the values (1) - (5):

  remoteControlPlane:
    enabled: true                         # < (1) set to true
    api:
      protocol: "http"                    # < (2) set protocol: http or https
      hostname: ""                        # < (3) set Keptn endpoint (without /api)
      apiValidateTls: true                # < (4 - optional) option to skip TLS verification
      token: ""                           # < (5) set Keptn API token
  

• Depending on your setup of the multi-cluster environment and the approach you modeled your staging process, one stage can be for example on a separate cluster. Let’s assume the following setup:

  • Project: sockshop

  • Service: carts

  • Stages:

    • hardening - on Cluster-A
    • production - on Cluster-B

    To properly configure the execution plane services that run, for example, on Cluster-A, the distributor in the values.yaml needs to be configured:

    distributor:
      projectFilter: ""                     # set the project, e.g., "sockshop" to get events for this project.
      stageFilter: "hardening"              # set the stage, e.g., "hardening" to get events for the stage.
      serviceFilter: ""                     # set the service, e.g., "carts" to get events for the service.
    

    Note: projectFilter, stageFilter, and serviceFilter allow a comma-separated list of values.

• Deploy the execution plane service (e.g., jmeter-service) from release assets with your values.yaml and by using helm:

  helm install jmeter-service https://github.com/keptn/keptn/releases/download/0.11.4/jmeter-service-0.11.4.tgz -n keptn-exec --create-namespace --values=values.yaml
  

• Test connection to Keptn control plane using:

  helm test jmeter-service -n keptn-exec
  

  Pod jmeter-service-test-api-connection pending
  Pod jmeter-service-test-api-connection succeeded
  NAME: jmeter-service
  LAST DEPLOYED: Thu Feb 25 15:55:24 2021
  NAMESPACE: keptn-exec
  STATUS: deployed
  REVISION: 1
  TEST SUITE:     jmeter-service-test-api-connection
  Last Started:   Thu Feb 25 15:55:40 2021
  Last Completed: Thu Feb 25 15:55:42 2021
  Phase:          Succeeded
  

How to uninstall an execution plane services?

• To uninstall an execution plane service, e.g., jmeter-service, just execute:

  helm uninstall jmeter-service -n keptn-exec
  

Summary of values to configure execution plane service

See the configuration parameters of the supported execution plane services:

• helm-service: Helm Chart values

• jmeter-service: Helm Chart values

The important once that are used in the above example are:

Troubleshooting

Execution plane service does not start working

If you see in the Keptn Bridge that an event was triggered but no service was reacting upon this trigger, test the connectivity from the execution plane service to the control plane. (as mentioned above) The Helm Charts for the helm-service and jmeter-service have a built in sanity check that validates whether the connection to the control plane can be established.

Test (sanity check):

Prerequisites:

• helm CLI

• Connect you to the cluster where the execution plane is running

• For example, you want to test jmeter-service that is running in keptn-exec namespace, execute:

helm test jmeter-service -n keptn-exec

* The expected outcome should be:

```console
Pod jmeter-service-test-api-connection pending
Pod jmeter-service-test-api-connection succeeded
NAME: jmeter-service
LAST DEPLOYED: Thu Feb 25 15:55:24 2021
NAMESPACE: keptn-exec
STATUS: deployed
REVISION: 1
TEST SUITE:     jmeter-service-test-api-connection
Last Started:   Thu Feb 25 15:55:40 2021
Last Completed: Thu Feb 25 15:55:42 2021
Phase:          Succeeded

Help:

1. Validate the properties set in the values.yaml:

remoteControlPlane:
enabled: true                         # < (1) set to true
api:
  protocol: "http"                    # < (2) set protocol: http or https
  hostname: ""                        # < (3) set Keptn hostname (without /api)
  apiValidateTls: true                # < (4 - optional) option to skip TLS verification
  token: ""                           # < (5) set Keptn API token

• Is enabled set to true?
• Is the Keptn API endpoint on http or https?
• Is the hostname of the Keptn API endpoint correct, e.g. my.keptn-dev.company.com (without /api)
• Do you want to skip TLS verification?
• Is the Keptn API token correct? (You can find it in the Keptn Bridge, or by following the guide for authenticating)