Microservices Testing: Feature Flags vs. Preview Environments

Originally posted on The New Stack.

Feature flags and preview environments are popular for managing microservices rollouts and testing. Learn the pros and cons for specific situations.

Microservices architectures have transformed how modern applications are built, allowing for rapid feature development and scalability. However, they also bring unique testing challenges. Ensuring that a new feature works as intended across multiple microservices without causing unintended breakages requires robust testing strategies. Two popular methods for managing feature rollouts and testing in microservices are feature flags and preview environments. But which approach is better?

Each has strengths and limitations, which I’ll explore through the lens of a fictional microservices application that is implementing a new ShoppingCart behavior and interacts with the Orders service, the Payment and a Wishlist service.

An example of an e-commerce service architecture. (From “An Open-Source Benchmark Suite for Microservices and Their Hardware-Software Implications for Cloud & Edge Systems,” ASPLOS ’19: Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems.)

Microservices and the Testing Challenge

In traditional monolithic applications, testing a new feature often involves verifying the entire application as a whole. In microservices, each service is developed, deployed and tested independently, making it harder to predict how changes in one service might affect others. For example, a small change to an authentication service could unexpectedly break the payment processor if their interaction isn’t tested thoroughly.

To ensure that such issues are caught early and before they impact users, testing strategies must evolve. This is where feature flags and preview environments come into play.

How Feature Flags Manage Feature Rollouts

Feature flags provide a dynamic way to manage feature rollouts by decoupling deployment from release. The basic idea is simple: New code is deployed to production, but it’s not active until the feature flag is turned on. This allows for gradual rollouts, targeted testing and easy rollbacks if something goes wrong.

For example, in our microservices app, we could implement a feature flag for the new ShoppingCart feature like this:

With this flag, we could enable the feature for a small percentage of users while testing its interaction with the Order, Payment and WishList services in a real-world environment.

Advantages of Feature Flags

1. Real-time testing in production: You can test features in the production environment, eliminating discrepancies between test and production. For example, monitoring how a new feature handles real traffic.
2. Controlled rollouts and quick rollbacks: You can roll out features gradually, limiting impact. If something breaks, you can disable it instantly, minimizing risks.
3. Separating deployment from release: You can deploy code separately from releasing it to users, ideal for aligning releases with specific events or milestones.

Things to Keep in Mind With Feature Flags

1. Complexity: Overuse adds complexity, multiplying configurations to test and making the system harder to manage.
2. No isolation: Flags run in production, so interactions between services can cause unexpected issues, like a bug in a payment processor.
3. Technical debt: Unremoved flags clutter the codebase, making maintenance harder. As one Reddit user noted, “Feature flags are a cancer unless removed quickly.”

Feature flags shouldn’t be used for every change. They are best suited for user-facing features, A/B tests or when real-time rollbacks are critical. But for deeper integration or infrastructural changes, relying solely on feature flags can be dangerous, as these types of changes often need comprehensive testing in isolated environments before reaching production. This is where preview environments shine.

How Preview Environments Enable Isolated Testing

A preview environment is an ephemeral, isolated environment that can be spun up on demand to test features in isolation. It can either clone the entire application or selectively deploy only the services being modified. Isolation is achieved either through duplicating the infrastructure or by dynamically routing specific requests to the services under test.

In our microservices app, a preview environment would let us test the new ShoppingCart’s integration with the rest of the services before deploying to production. We could run full integration tests that simulate real-world user scenarios, catch bugs early and iterate without affecting the live system.

Advantages of Preview Environments

1. Isolation: Unlike feature flags, preview environments isolate the new feature, reducing the risk of it affecting other services. For example, any bugs found in the Cart Service wouldn’t affect other teams testing other parts of the application.
2. Early bug detection: By testing in an environment that closely mimics production, you can catch bugs early, reducing the risk of introducing issues postdeployment. This is especially important in complex microservices architectures, where service interactions can be hard to predict.
3. Confidence before merging: Preview environments allow comprehensive testing premerge, ensuring that the feature is fully tested across all services before it reaches production.

Things to Keep in Mind With Preview Environments

1. Resource usage: Cloning an entire infrastructure is resource-heavy, especially for large apps. Newer approaches focus on isolating modified services, reducing overhead.
2. Developer experience: Setup time varies. Full infrastructure cloning is slower, while dynamic routing for specific services is faster and less burdensome.
3. Traffic simulation: Preview environments help with early testing but may not fully mimic real traffic. Dynamic routing can better simulate production conditions using a high-fidelity preproduction setup.

Each of these considerations depends on the implementation approach — whether isolating full infrastructure or just the services being changed in a preproduction environment.

Combining Feature Flags and Preview Environments

While both feature flags and preview environments offer distinct advantages, the reality is that neither is perfect on its own. The best approach for testing microservices is often a combination of both.

Here’s a practical approach:

• Use preview environments for premerge testing: Spin up a preview environment to fully test the new recommendation algorithm’s integration with other services. This ensures that you catch any bugs before the code is merged into the main branch and deployed to production.
• Use feature flags for postdeployment testing: Once the feature passes integration testing in the preview environment, deploy it to production behind a feature flag. Gradually roll it out to a small subset of users, monitor its behavior under real-world conditions and roll back if any issues arise.

This hybrid approach allows you to leverage the strengths of both methods: the early bug detection and isolation of preview environments, combined with the flexibility and real-time testing of feature flags.

A Balanced Strategy for Microservices Testing

Effective microservices testing requires balancing speed and reliability. Feature flags enable real-time testing in production but often lack isolation for complex integration issues. Preview environments offer isolation for premerge testing but can be resource-intensive and may not fully replicate production traffic.

The best approach? Combine both. Use preview environments to catch bugs early, and then deploy with feature flags to control the release in production. This ensures speed without sacrificing quality.

At Signadot, we help companies create and manage cost-efficient preview and ephemeral environments. Companies like Earnest and Brex use our lightweight Sandboxes to streamline testing. Learn more about how we can help.