Skip to main content

Run Automated Tests on PRs

Overview​

As development teams iterate on microservices independently, it is important to ensure that relationships between services are healthy with each change being made. Running Integration & E2E tests early helps in ensuring that any critical flows are tested early in the development lifecycle.

In the absence of such testing, there are often issues that are discovered late in the development lifecycle causing long feedback loops and wasted development time in finding and resolving issues that could be caught much earlier.

High-Level Steps​

1. Create Pull Request Sandboxes​

Follow the steps outlined in the Pull Request Sandboxes guide to create a sandbox for each pull request. In the rest of the guide, we will set up automated tests to run on top of these Pull Request sandboxes.

2. Create a Job Runner Group​

A Job Runner Group creates pods to run your tests. You can use your own docker images and pod templates. These pods will be created in the cluster and are pre-warmed to speed up the execution of jobs.

name: my-jrg
spec:
  cluster: "my-cluster"
  labels:
    env: "test-env"
  namespace: my-tests-ns      # namespace where executors will live
  image: runner/image:latest  # runner imae
  jobTimeout: 30m             # jobs can run for a maximum to 30m
  scaling:
    manual:
      desiredPods: 2          # creates 2 pods, can run 2 jobs in parallel

This above specification defines a Job Runner Group named my-jrg which uses a custom runner image. You can create my-jrg directly in the UI, or using the CLI. An example Job Runner Group specification for running cypress tests is shown here.

3. Create a Job Template to Run Tests​

Next, set up a job template that takes a script to run the tests. Here’s an example configuration:

spec:
  namePrefix: my-job
  runnerGroup: my-jrg # which runner group to run within
  script: |
    #!/bin/bash
    set -e

    # Clone the git repo
    echo "Clone repo"
    git clone --single-branch -b "@{branch}" \
      https://github.com/org/repo.git

    # Run the test
    cd tests/
    ./run-tests.sh
    
  routingContext:         
    sandbox: "@{sandbox}" # controls the injection of routing env vars into the above script
  uploadArtifact:         # uploading artifacts
    - path: path/output.mp4

There are other fields as well in the Job Runner Group that allow you to specify a custom Pod Template to attach secrets / credentials to the runner pods so that they are available to the jobs for say - cloning Git repositories. You can learn more about all the different options available in the Job Runner Group reference.

Note how this has two template variables, the branch and the sandbox. You can now supply these parameters at run-time as shown:

signadot job submit -f ./job-template.yaml --set branch=main --set sandbox=my-sandbox --attach

This job template can be used with the signadot job submit command and each submission will have a distinct name that is derived based on the namePrefix field. An example Job template for running cypress tests is shown here.

3. Using the Routing Key in Tests​

Within your test, ensure you use the routing key to target the Sandbox environment. There are the following environment variables available within the script of the Job specification:

  • SIGNADOT_ROUTING_KEY
  • SIGNADOT_SANDBOX_NAME

You can use the above parameters and pass them into your test, so that the tests can inject the appropriate routing key into requests they are making so that it can target the sandbox correctly.

4. Integrating with CI Pipeline​

Finally, integrate the jobs with your CI pipeline. You'll typically run jobs after creating the sandbox from your specific CI/CD pipeline using the signadot CLI. For detailed instructions, refer to the guide on integrating with CI/CD.

Best Practices​

Storing Specifications​

Creating a Job Runner Group is typically a one-time activity, so, it could be specified GitOps style in a central repository, or created from the UI and managed entirely within Signadot.

The Job Template is typically stored in Git. Some of the common locations where job specifications are typically stored include:

  • Source control / git within a monorepo (example)
  • Source control / git in a central infrastructure repository (typically in a multi-repo setup)

In source control, they may take the following structure to store and maintain multiple different job specs as needed:

- signadot/job-integration-routeservice-mocha.yaml
- signadot/job-e2e-routeservice-cypress.yaml
...

Choosing the Image for a Job Runner Group​

When choosing the Docker image for a Job Runner Group, you have a lot of flexibility depending on your specific needs and the nature of the jobs you are running.

  1. Generic Upstream Docker Image: You can start with a general upstream Docker image like ubuntu and choose to install dependencies in the job's script and run tests on it. This approach helps reuse the same runner group across many types of jobs without needing to create or maintain any new docker images.

  2. Custom Docker Images: As you optimize your tests, you may choose to create custom Docker images. In these images, you can bake in all the testing dependencies. This can significantly speed up the tests as the dependencies do not need to be installed every time a job is run.

We recommend starting with the general image. As you understand your testing needs better, you can create custom Docker images to optimize tests and make them run faster.

Conclusion​

This guide provides a step-by-step process for setting up Automated tests for sandboxes, making it easy for users to implement this in your development workflow.