Skip to main content

Smart Test Specification

Overview​

info

This feature is in beta. Please leave feedback via our community GitHub or Slack.

  • This feature requires Signadot Operator v0.19+.
  • You must enable Managed Runner Group on your cluster via the Signadot Dashboard (under Settings) and have at least 1 runner pod.

Signadot tests are API integration test specifications designed to help you maintain robust and reliable tests. These tests are automatically executed on both sandbox and baseline environments to ensure stability and prevent flakiness. They currently support contract testing via traffic capture & diff as described below.

Test Specification​

Tests are specified in Starlark which is a language similar to Python3. Code is run using starlet which provides several libraries. Detailed documentation of the libraries can be found here.

In addition to the starlet libraries, we extend the http module to accept two optional flags for each request: capture (type bool) and reqName (type string).

For example:

resp = http.get("...", capture=True, name="getLocations")

When capture is true, the library will inject headers based on environment variables to interface with our system of traffic capture. In this case, one can name the request with reqName.

Am example Smart Test is shown below:

res = http.get(
    url="https://my-api:8081/locations", # can also be cluster internal URLs like http://name.namespace.svc
    capture=True, # enables Smart Diff
    name="getLocations"
)

print(res.status_code)
print(res.body())

This test runs on a sandbox and a baseline, and presents a Smart Diff of the requests and responses received in both of these contexts. The requests are modeled so that the most relevant differences are highlighted. You can read more about how that works in the concept docs. The result is presented similar to the below via the Signadot Dashboard.

Smart Diff Results

Triggers​

Triggers define which sandboxes will trigger tests and how the tests are executed. A single test can have many triggers.

Triggers are specified via the Signadot Dashboard using the "Triggers" Tab after you create a Smart Test. Triggers currently allow matching sandboxes based on the baseline workload being sandboxed. This means that if you designate a specific cluster and workload as a trigger, the test will automatically run on any sandbox that includes that workload whenever it is created or updated.


In the above example, whenever you create or update a sandbox that contains the workload location in the namespace hotrod-istio, the test will be automatically triggered on that sandbox.

Known Limitations​

Currently, Smart Tests are limited to handling RESTful APIs with JSON payloads. This means that other types of traffic, such as gRPC, SOAP, or non-JSON data formats, are not supported at this time. Future updates may expand the scope of supported protocols and data formats.