Skip to main content

Smart Tests and Jobs

Overview​

Signadot has two distinct provisions for testing microservices in Kubernetes.

  1. Smart Tests which are native tests designed for integration testing of APIs. They leverage advanced features like traffic capture and modeling, and support integration testing that the system helps manage and maintain over time. With Smart Tests, users can easily access features such as automatic comparison of sandbox to baseline traffic.

  2. Jobs which comprise infrastructure support for running your test suites using any test framework in your Kubernetes clusters.

Smart Tests​

Smart Tests are Integration Tests authored in Starlark that automatically run on the baseline and sandbox. These tests can be used for contract and API integration testing of microservices running in Sandboxes. Under the hood, Smart Tests uses Jobs. For more details on how to use Smart Tests, refer to the Smart Test reference.

Smart Diff​

One feature of Smart Tests is the ability to capture requests and responses between microservices, and to compare those requests and responses between the baseline and sandbox with noise filtering.

Sending requests in a test with the option capture=True will cause that request and response to be captured and sent to a traffic model service on our Control Plane for baseline traffic. Additionally, any downstream request hitting the sandbox will be captured and modeled.

The traffic modeling uses language models of structured data to record and predict the probabilities of various structures. Running a Smart Test will produce a diff in which every operation is given a relevance score in [0..1] and categorized into Red, Yellow, and Green relevance. Highly relevant diff operations (Red) are likely field additions or removals, structural changes, or unlikely changes in field values. Low relevance diff operations (Green) reflect noisy changes which are likely in any event independent of the sandbox. Those which fall within (Yellow) may be worth review.

Jobs​

Jobs are run in pre-warmed pods in your kubernetes cluster on an image and in an environment defined by a Job Runner Group. The standard output and error can be streamed to CI and artifacts can be produced and subsequently downloaded. For more details on how to use Jobs, refer to the Job reference, and the Job Runner Groups reference.

When Jobs are run in the context of a Sandbox or Route Group, the system monitors the sandbox or routegroup for readiness during the execution of the Job to help ensure its results are coherent with the Sandbox or Route Group.

Jobs and Testing​

Jobs are ideal for testing using existing test suites and frameworks. They are an infrastructure level concept for testing in Kubernetes and do not attempt to support any specific kind of testing per se.