Skip to main content

signadot smart-test

Note

The examples here work with v0.9.0+ of the Signadot CLI.

The CLI provides a smart-test command (alias: st) which can be used to run Smart Tests stored in a git repository or in files and also to examine, query, or cancel individual runs of Smart Tests.

This documentation is organized into the following sections:

  • signadot smart-test run - Running smart tests
  • signadot smart-test execution - Managing test executions
  • Managing Repository Configuration - Configuring test discovery and labels

signadot smart-test run​

The run command can be used to run smart tests for an entire repository, or in a directory or file. All such smart tests are expected to be in files with suffix .star and to contain smart test source code.

When using signadot smart-test run one

  • Specifies where to run the tests by either:
    • specifying a Signadot cluster with the --cluster flag to run on baseline; or
    • specifying a Sandbox or RouteGroup routing context to run tests and additionally enable Smart Diff analysis of captured API traffic
  • Specifies what tests are run by
    • using repository configuration of the working directory (see Managing Repository Configuration section for details); or
    • running all tests in a directory, recursively, using the -d flag; or
    • running a specific file using the -f flag.
  • Specifies whether or not the run of the test is published, meaning it shows up by default on the dashboard and in associated pull requests, using the --publish flag. The default is to not publish the test run.
  • May specify any additional labels to attach to the tests to run with the --set-label flag.
  • Specifies the behavior of the command with respect to its exit code and whether or not it waits for tests to complete.
  • Specifies the output format.
  • Generates a unique identifier for all tests run with the invocation.
signadot smart-test run [flags]

Every invocation generates a unique identifier called a run ID. Run IDs are random strings which sort in an order consistent with the time ordering over invocations. These IDs provide a handle on all test executions created for a given invocation.

Run IDs are available in both unstructured and structured output:

# Unstructured output
signadot smart-test run --cluster minikube
Created test run with ID "ei8hd6x38tts" in cluster "minikube".
...

# Structured output
signadot smart-test run --cluster minikube -o yaml --no-wait | yq .runID
ei8higamdm9s

signadot smart-test execution​

The smart-test execution command (alias: x) allows canceling, listing and retrieving individual executions:

signadot smart-test execution [command]

Available Commands:
  cancel      Cancel a test execution
  get         Get a test execution
  list        List test executions

Test executions are identified by the run ID followed by a -N where N is an increasing number, one for each test execution created with the run.

Common usage:

# List tests from a run
signadot smart-test execution list --run-id <run-id>

# Get details of a specific test
signadot smart-test execution get <execution-id>

# Cancel tests from a run
signadot smart-test execution cancel --run-id <run-id>

Managing Repository Configuration​

One can configure a git repository to indicate what smart tests to run by default if signadot smart-test run is invoked inside a that repository.

File Structure​

Here is an example repository structure:

% cd Dev/my-repo
% git status
On branch main
...
% cat .signadot/config.yaml
smart_tests:
- tests/backend
- tests/frontend
% tree tests
├── .labels
├── backend
│   ├── .labels
│   ├── team1
│   │   ├── .labels
│   │   └── file1.star
│   ├── team2
│   │   ├── .labels
│   │   └── file2.star
│   └── test.star
├── frontend
│   ├── .labels
│   └── file3.star
├── ignore.star  # not run because not in the config.yaml
└── ignored      # not run because not in the config.yaml
    └── file4.star

Labels​

To help organise test executions, one can associate labels with tests using the filesystem or the --set-label flag.

In the filesystem, one can add a .labels file in any directory within the repository root to apply the label to all test executions of any .star file located under that directory. If run outside a repository, labels are taken only from the directory containing a test.

The .labels files contain key=value pairs, one per line:

% cat tests/fixtures/smart-tests-git/.labels
suite=integration

% cat tests/fixtures/smart-tests-git/backend/.labels
area=backend

To set a label at invocation time:

% signadot smart-test run --set-label pr=122