Shifting Left with Signadot: How Earnest Empowered Developers for Early Testing

Enabling more reliable release process and a better Developer Experience

Fintech company Earnest is a technology-lender that brings low-interest loans to high-potential people. With both a platform engineering and developer experience team, Earnest maintains focus on a secure, efficient release process with as few surprises as possible. With Signadot, the Developer Experience (DevEx) team was able to automate end-to-end and integration tests. This gave engineers better insight into how their services are running during a test, and caused more reliable commits to Staging.

I recently got the chance to sit down with Early Ehlinger, lead on the Earnest DevEx team, and Daniel Jimenez, Infrastructure DevEx Engineer, to talk about their journey to test automation with Signadot. 

Why Earnest needed Signadot  

As we at Signadot have written in the past, testing and experimentation can become quite difficult as microservice architecture gets more complex. When you don’t have reliable, high-accuracy integration tests before pushing to staging, the result is more out-of-band bug fixes as your team tries to diagnose problems.

Without Signadot, developers at Earnest didn’t have ready access to testing Sandbox that included up-to-date dependencies for their service. This meant that releases were often handed off without 100% confidence that the code would pass integration testing. This meant having a separate team identify the problem, try to diagnose it, and then reach out to the original product team with a description of the problem. This process, with several points of human-to-human communication, added friction and hurt developer velocity.

“Signadot is a key part of our solution for the most critical parts of our testing pyramid. For integration and end-to-end testing we need to have a sandbox up and running because they need a live service to talk to.” - Early Ehlinger, Dev Experience Lead

Signadot uses requests isolation to let multiple teams use a single high-fidelity cluster for testing and to try out new features, without their changes impacting other teams. This ‘sandbox isolation’ is a core benefit of using Signadot. For Earnest, their first use of Signadot was to allow users to create sandboxes manually: designating the services that the engineer wanted to ‘fork’ and letting this forked service interact with the baseline environment.

Moving to automated Sandboxes

‍

From manual sandboxes, the Developer Experience team found it easy to move to an automated system:

“We’d been trying Signadot, letting people manually create sandboxes, for a while so switching over to automatically creating sandboxes and running tests was a pretty smooth process” - Daniel Jimenez

‍

The team was highly motivated to improve the performance of integration testing. As Early put it: “we were looking for what can we do from an infrastructure perspective without going in and rewriting all the tests to make them fast? Previously we were parallelizing tests, but we're doing it by hand. So you'd still end up with chunks of like 10 or 20 tests that would run in series and so they would take like, you know, 30 minutes to an hour. Whereas, the longest run on any individual test might have been 10 minutes.”

This led the team to migrate Testkube so that the same tests, with the same execution time, could run faster by running in parallel. Early goes on to describe the advantages:

“What’s nice about Testkube is you can very easily set things up so that an individual source file in your Cypress repo or your Mocha repo or whatever, correlates to a single test in Testkube. If you design things right, we can run all of the tests in a suite in parallel. If you parallelize it, that whole chunk takes only 10 minutes instead of an hour. And we saw really drastic improvements in terms of speed just by letting something run everything in parallel for us.”

Once that work was done, all that was needed to integrate Signadot was to make sure the routing key was available for the test suite, so that it could supply the appropriate request header while testing.

“It was pretty easy to integrate it with Signadot. The only thing we had to do really was make sure we got environment variables for the routing key plumbed through all the different stages.”

Teams leverage Cypress to automate end-to-end testing of applications deployed in Kubernetes clusters, ensuring each component interacts seamlessly across the orchestrated environment. The team configured adding signadot sandbox headers to each Cypress test invocation and running those tests within Kubernetes.

‍

The goal for any advanced user of Signadot is for every commit to a branch to be easily flagged to create a sandbox based on these changes. At Earnest, the Developer Experience team was able to implement all the setup required.

‍

“Our Dev Ex team is able to maintain all the test setup and sandbox spec in our Jenkins library. With a standard naming convention, you can automatically flag a deploy to create a sandbox for testing” Early Ehlinger, Dev Experience Lead

Testing the critical paths

For fintech companies like Earnest, the steps a user takes on a path to conversion can be extremely high-stakes. Earnest helps students apply for financial services, and the steps along that path need to work with every deployment. Daniel Jimenez describes the process of ensuring functionality with integration tests:

‍

“Basically an integration test goes through the flow of any of our products and emulates a user interaction. There are tests to emulate every critical part of our flow, such as document signing. There’s a test to create a [loan] application, and check to see if the applicant was approved or not, whatever it's expecting. Then a separate test that will go into an existing application, see that an offer has been made, and verify that you get a .pdf and that you can sign it.”

One of the key benefits of Signadot is the amount of insight offered into the services under tests. Daniel continues:

‍

“People really like Signadot. They like being able to go in and see their service’s logs, and they find it really useful to go in and see the errors from their service during a test, catching problems before they emerge on Staging.”

‍

While any organization can take the time for extensive testing on Staging to ensure that everything functions properly, the key advantage of a Signadot-enabled process is that it's the developers who are aware of a problem first, before their code reaches staging. This means that the person working on the feature has direct knowledge of the problem and can work on a fix directly, rather than waiting for later feedback from a software testing team.

‍

Front-end focused testing with Signadot

Signadot isn’t just a tool for backend developers maintaining a microservice, it also helps developers working closer to the frontend to preview changes that involve one or multiple updated microservices. Kai Rillet, a software engineer at Earnest, lead an effort to get more developers to preview changes using signadot’s routing key right from their browser.

By adding a header with a simple browser plugin, any developer can preview a sandbox based on their or another developer’s branch. This ability to preview frontend changes with all the normal dependencies hosted in a shared cluster means that developers can be more confident their changes interact with every other service correctly. 

Giving testing power to the developers

While the initial setup of the Signadot integration and end-to-end tests was completed by the developer experience team, the Earnest engineering org would like to move toward a model where developers manage these more complete tests directly.

“Developer Experience had a really easy time setting up the integration and migrating our current suite of integration tests to using Signadot. As an org we want to migrate to having the developers maintain these. Individual developers don’t mind writing unit tests, but it requires a cultural shift to have them write and maintain integration tests.” Early Ehlinger, Dev Experience Lead

However this breakdown of work develops, Signadot will remain a key component of the process for creating highly accurate test environments to test new changes.

Conclusion: meeting testing and reliability changes

‍

Earnest's integration of Signadot into their release process represents a significant leap forward in the realm of automated testing and reliable deployment strategies. By leveraging Signadot's capabilities for creating isolated sandboxes and automating end-to-end and integration tests, Earnest has not only streamlined their development workflow but also enhanced the reliability and security of their releases. This approach has empowered developers with direct insights into their service's performance during tests, enabling quicker identification and resolution of issues before they escalate to staging environments.

The shift towards automated sandbox creation and the emphasis on testing critical user paths reflect a proactive approach to software development that prioritizes user experience and system reliability.

The lessons from Earnest's experience with Signadot offer valuable insights for other organizations striving for excellence in their development practices. The move towards giving developers greater responsibility for integration testing signifies a broader cultural shift towards ownership and accountability in software engineering. In this context, tools like Signadot are not just facilitating more efficient testing processes; they are reshaping the very dynamics of team collaboration and product development.

‍