Quality Gates evaluations

A quality gate allows you to validate a deployment or release using data provided by your observability platform and using quality criteria that you define as Service Level Objectives (SLOs); see the SLO reference page.

ℹ️ If you are not familiar with the concept of a quality gate, learn more here.

Video demonstration

The following video illustrates how to implement a quality gate evaluation:

Service Level Providers, SLIs and SLOs in Keptn

This shows how to configure the three elements that Keptn requires to run an evaluation:

• The observability platform to use as a data source
• Which metrics (SLIs) to retrieve from that observability platform
• What threshholds (SLOs) are acceptable for each metric

You will also see

• How to apply weights to SLIs
• How to apply relative threshholds to SLOs for regression testing
• How to use the key SLI field to identify a metric that must pass for a build to pass, no matter what the results of the other SLOs are.

Definition of project, stage, and service

Let’s assume you have an application that is running in an environment and composed of one or multiple services (aka. microservices). For example, you have an application called easyBooking, which can be broken down into the booking and payment service. Besides, the application is running in a quality_assurance environment (aka. stage). In order to manage your application and services in Keptn, you need a Keptn project, stage and services. For the easyBooking application, the Keptn entities of a project, stage, and service map to the example as follows:

• project: easyBooking
• stage: quality_assurance
• service: booking & payment (For the sake of simplicity, a quality gate will be configured for booking only.)

For defining the stage(s) a service has to go through, a Shipyard file is needed. Since a quality gate should be configured for the quality_assurance environment only, the corresponding Shipyard for the easyBooking project looks as follows:

apiVersion: "spec.keptn.sh/0.2.3"
kind: "Shipyard"
metadata:
  name: "shipyard-sockshop"
spec:
  stages:
    - name: "quality-assurance"

You do not have to define any task sequence in a stage because quality gates (aka. evaluation) are a built-in Keptn capability. Hence, there is no need to explicitly add an evaluation sequence. However, the explicit form of the above Shipyard file would look as the following one, which behaves the same way:

apiVersion: "spec.keptn.sh/0.2.3"
kind: "Shipyard"
metadata:
  name: "shipyard-sockshop"
spec:
  stages:
    - name: "quality-assurance"
      sequences:
       - name: "evaluation"
         tasks:
         - name: "evaluation"

Create project and service

• To create the Keptn project easyBooking, use the keptn create project command as discussed in Create a project.

• To create the Keptn service booking, use the keptn create service command as discussed in Create a service.

Note: It is not necessary to create the stage since it is declared in the Shipyard that is applied during the project creation.

Step 1: Create project and service

• To create the Keptn project (e.g., easyBooking), use the keptn create project CLI command. Here the Shipyard file is needed, which declares the stages.

• To create the Keptn service (e.g., booking), use the keptn create service CLI command.

Step 2: Configure Keptn to use your SLI-provider and add SLIs

Depending on the monitoring solution you have in place, a corresponding SLI-provider is required by Keptn. This SLI-provider gets its queries for the individual metrics from the Service-Level Indicator (SLI) config.

• Familiarize yourself with the concept of an SLI and derive the set of indicators required for, e.g., the booking service.

• Follow the steps of deploying an SLI-provider and uploading an SLI config as described on the reference page for the monitoring service you are using as a data source:

• Datadog

• Dynatrace

• Prometheus

Step 3: Add SLO configuration to a service

By adding an Service-Level Objective (SLO) config to your service, you activate a quality gate for that service.

• To add an SLO to the booking service, use the keptn add-resource CLI command.

Step 4: Use the quality gate

At this point, you have:

✔️ created the project e.g. easyBooking and the service e.g. booking

✔️ configured Keptn to use your SLI-provider with custom SLIs

✔️ activated the quality gate for e.g. booking by providing an SLO

To trigger a quality gate evaluation, execute the keptn trigger evaluation CLI command. This CLI command sends an event to Keptn, which acknowledges receipt of the event by returning a unique ID (keptn-context). This unique ID is required to fetch the result of the quality gate evaluation. This is discussed more below.

Working with quality gates

In order to work with and integrate quality gates, two actions are needed:

1. Trigger a quality gate evaluation for a service in a specific project/stage and for a certain timeframe
2. Fetch the evaluation result of a quality gate

This section explains how to use the Keptn CLI and the Keptn API for those two actions.

Use the Keptn CLI to work with quality gates

• To work with the Keptn CLI, make sure your Keptn CLI is authenticated against your Keptn installation.

• To verify the CLI connection to your Keptn, execute keptn status.

Trigger a quality gate evaluation

To trigger a quality gate evaluation for a service in the stage of a specific project, the Keptn CLI provides two commands:

• The keptn trigger evaluation command allows specifying the timeframe of the evaluation using the --start, --end, or timeframe flags.

• To trigger a quality gate evaluation with a timeframe of 5 minutes starting at 2020-12-31T10:00:00, use the following example:

keptn trigger evaluation --project=easyBooking --stage=quality_assurance --service=booking --start=2020-12-31T11:59:59 --timeframe=5m

• This command returns a unique ID (keptn-context) that is required to retrieve the evaluation result.

• First, specify a valid Keptn CloudEvent of type sh.keptn.event..evaluation.triggered and store it as JSON file, e.g., trigger_quality_gate.json. Choose one of the following three options to specify the event:

  • Option 1 - Define start and end time to evaluate:

    {
        "source": "keptn-cli",
        "specversion": "1.0",
        "type": "sh.keptn.event.quality_assurance.evaluation.triggered",
        "contenttype": "application/json",
        "data": {
          "evaluation": {
            "start": "2020-09-01T08:31:06Z",
            "end": "2020-09-01T08:36:06Z"
          },
          "labels": {
            "buildId": "build-17",
            "owner": "JohnDoe",
            "testNo": "47-11"
          },
          "project": "easyBooking",
          "service": "booking",
          "stage": "quality_assurance"
        }
    }
    

  • Option 2 - Define time frame to evaluate

    {
        "source": "keptn-cli",
        "specversion": "1.0",
        "type": "sh.keptn.event.quality_assurance.evaluation.triggered",
        "contenttype": "application/json",
        "data": {
          "evaluation": {
            "timeframe": "5m"
          },
          "labels": {
            "buildId": "build-17",
            "owner": "JohnDoe",
            "testNo": "47-11"
          },
          "project": "easyBooking",
          "service": "booking",
          "stage": "quality_assurance"
        }
    }
    

  • Option 3 - Define start and end time of previous test

    {
        "source": "keptn-cli",
        "specversion": "1.0",
        "type": "sh.keptn.event.quality_assurance.evaluation.triggered",
        "contenttype": "application/json",
        "data": {
          "test": {
            "start": "2020-09-01T08:31:06Z",
            "end": "2020-09-01T08:36:06Z"
          },
          "labels": {
            "buildId": "build-17",
            "owner": "JohnDoe",
            "testNo": "47-11"
          },
          "project": "easyBooking",
          "service": "booking",
          "stage": "quality_assurance"
        }
    }
    

• Trigger a quality gate evaluation by sending the CloudEvent to Keptn using the keptn send event command:

keptn send event --file=trigger_quality_gate.json 

• This command returns a unique ID (keptn-context) that is required to retrieve the evaluation result.

Fetch the evaluation result of a quality gate

• To fetch a quality gate evaluation result, the Keptn CLI provides the keptn get event sh.keptn.event.evaluation.finished command. This command requires the ID (--keptn-context) returned by the trigger evaluation or send event command, as well as the name of the project (--project).

keptn get event sh.keptn.event.evaluation.finished --keptn-context=1234-5678-90ab-cdef --project=easyBooking

Note: The evaluation of the quality gate may take some time, especially because the metrics first have to be available in your monitoring solution and need to be queried by the SLI-provider. Consequently, the result is not immediately available.

Use the Keptn API to work with quality gates

• To work with the Keptn API, get the API token from the Keptn Bridge and follow the Keptn API link to the Swagger-UI.

• If you want to interact with the Keptn API via cURL, you also need the Keptn API URL and API token

Trigger a quality gate evaluation

To trigger a quality gate evaluation for a service in the stage of a specific project, the Keptn API provides two endpoints:

• This endpoint requires the path parameters projectName, stageName, and serviceName: /api/v1/project/{projectName}/stage/{stageName}/service/{serviceName}/evaluation

• The required payload allows you to set the start, end, and timeframe (choose either the end or timeframe parameter):

{
    "start": "2020-09-28T07:00:00",     // required
    "end": "2020-09-28T07:05:00",       // cannot be used in combination with 'timeframe'
    "timeframe": "5m",                  // cannot be used in combination with 'to'
    "labels": {
      "buildId": "build-17",
      "owner": "JohnDoe",
      "testNo": "47-11"
    }
}

• Trigger a quality gate evaluation with a POST request:

curl -X POST "${KEPTN_ENDPOINT}/v1/project/easyBooking/stage/quality_assurance/service/booking/evaluation" \
-H "accept: application/json; charset=utf-8" \
-H "x-token: ${KEPTN_API_TOKEN}" \
-H "Content-Type: application/json; charset=utf-8" \
-d "{ \"start\": \"2020-09-28T07:00:00\", \"timeframe\": \"5m\", \"labels\":{\"buildId\":\"build-17\",\"owner\":\"JohnDoe\",\"testNo\":\"47-11\"}}"

• The endpoint returns a unique ID (keptn-context) that is required to retrieve the evaluation result. (Note: The response also contains a token that is required to open a WebSocket communication. This token is not needed now.)

• Specify a valid Keptn CloudEvent of type sh.keptn.event.[stage].evaluation.triggered and store it as JSON file, e.g., trigger_quality_gate.json. Choose one of the following three options to specify the event:

  • Option 1 - Define start and end time to evaluate:

    {
        "source": "keptn-cli",
        "specversion": "1.0",
        "type": "sh.keptn.event.quality_assurance.evaluation.triggered",
        "contenttype": "application/json",
        "data": {
          "evaluation": {
            "start": "2020-09-01T08:31:06Z",
            "end": "2020-09-01T08:36:06Z"
          },
          "labels": {
            "buildId": "build-17",
            "owner": "JohnDoe",
            "testNo": "47-11"
          },
          "project": "easyBooking",
          "service": "booking",
          "stage": "quality_assurance"
        }
    }
    

  • Option 2 - Define time frame to evaluate

    {
        "source": "keptn-cli",
        "specversion": "1.0",
        "type": "sh.keptn.event.quality_assurance.evaluation.triggered",
        "contenttype": "application/json",
        "data": {
          "evaluation": {
            "timeframe": "5m"
          },
          "labels": {
            "buildId": "build-17",
            "owner": "JohnDoe",
            "testNo": "47-11"
          },
          "project": "easyBooking",
          "service": "booking",
          "stage": "quality_assurance"
        }
    }
    

  • Option 3 - Define start and end time of previous test

    {
        "source": "keptn-cli",
        "specversion": "1.0",
        "type": "sh.keptn.event.quality_assurance.evaluation.triggered",
        "contenttype": "application/json",
        "data": {
          "test": {
            "start": "2020-09-01T08:31:06Z",
            "end": "2020-09-01T08:36:06Z"
          },
          "labels": {
            "buildId": "build-17",
            "owner": "JohnDoe",
            "testNo": "47-11"
          },
          "project": "easyBooking",
          "service": "booking",
          "stage": "quality_assurance"
        }
    }
    

• Trigger a quality gate evaluation with a POST request on /event:

curl -X POST "${KEPTN_ENDPOINT}/v1/event" \
-H "accept: application/json; charset=utf-8" \
-H "x-token: ${KEPTN_API_TOKEN}" \
-H "Content-Type: application/json; charset=utf-8" \
-d @./trigger_quality_gate.json

• The endpoint returns a unique ID (keptn-context) that is required to retrieve the evaluation result. (Note: The response also contains a token that is required to open a WebSocket communication. This token is not needed now.)

Fetch the evaluation result of a quality gate

• To fetch a quality gate evaluation result, the Keptn CLI provides the /event endpoint. This endpoint requires the query parameters keptn-context and type; latter is always sh.keptn.event.evaluation.finished.

curl -X GET "${KEPTN_ENDPOINT}/api/mongodb-datastore/event?keptnContext={keptnContext}&type=sh.keptn.event.evaluation.finished" \
-H "accept: application/json; charset=utf-8" \
-H "x-token: ${KEPTN_API_TOKEN}"

Note: The evaluation of the quality gate may take some time, especially because the metrics first have to be available in your monitoring solution and need to be queried by the SLI-provider. Consequently, the result is not immediately available.

Integrate into an existing pipeline

To integrate quality gates into an existing pipeline, use the API-based approach outlined above. As stated there, the evaluation result is not immediately available. Hence, build your integration using a polling mechanism that polls the evaluation result every 10 seconds and terminates after, e.g., 10 retries.

Extract evaluation results

Evaluation results can be extracted then forwarded to a third-party service for storing and processing. For example, you may want to push the results to Slack or another chat tool.

This can be implemented using either Webhooks or the Keptn API:

• Webhooks

  1. Use the Webhook service
     • See the Slack Webhook Integration example for more detailed instructions.
  2. Subscribe to the evaluation.finished event.
  3. Push the result to a webhook on your external system.

• Keptn API

  1. Use the Keptn API as discussed above.
  2. Query the evaluation.finished events for a given project, specifying the timeframe. For example, run a cron job every hour and query the last hour.
  3. When a new result is found, push it to your external system.

In most cases, the Webhook implementation is the better choice because it immediately pushes the result to the third party system rather than waiting for the scheduled cron job to run. The Keptn API implementation is mostly useful when the external service does not provide webhooks.