How Developers at DoorDash get 10x faster feedback on code changes

DoorDash Overview

DoorDash is a technology company that connects consumers with their favorite local businesses in more than 25 countries across the globe. Founded in 2013, DoorDash builds products and services to help businesses innovate, grow, and reach more customers.

DoorDash employs a multi-tenant architecture consisting of 100s of microservices operating on Kubernetes. Due to their on-demand service, the company's extensive developer team must innovate rapidly while ensuring minimal disruption for customers. To achieve this, DoorDash leverages end-to-end feature testing on their production environment.

Before Signadot

Spinning up new Docker images for changes proved to be a time-consuming process, taking more than 30 minutes for each deployment. Additionally, if a team introduced a new change with regressions, staging would break. This significantly impacted other teams waiting for the change to be fixed or rolled back, causing further delays and inefficiencies. Relying on unit and mocked CI tests proved inadequate since they did not address the dynamic end-to-end functionality. Furthermore, developers wanted greater control over the specific user flows that required testing.

Maintaining the accuracy and reliability of the staging environment independently proved to be a challenging task at the scale the team was operating. The difficulties in ensuring the fidelity of the staging environment made it harder for the developers to accurately test their changes before deploying them to production. These challenges created a bottleneck in the development process and hindered the team's ability to deliver features quickly and efficiently.

After Signadot

Signadot's vision of providing developers with early access to high fidelity testing in the development workflow aligned with DoorDash’s long term vision. Recognizing the potential benefits, DoorDash made the strategic decision to remove the staging environment entirely and adopt Signadot's approach. As a result, developers at DoorDash now have the ability to safely test their changes directly in the production environment from their laptops. This shift allows for more efficient and streamlined testing, enabling developers to iterate and refine their work with greater speed and accuracy.

How Developers use Signadot

DoorDash developers make code changes to their microservice(s) locally on their workstations, creating a "Local Sandbox" environment that has a unique identifier. In order to test these changes end-to-end in production, developers access the Web front end and utilize a Chrome Extension to specify the identifier associated with their Local Sandbox. Signadot ensures that these end-to-end flows are routed to the correct local workstation based on the unique identifier.

When it comes to testing from the mobile App, they use a debug version of the mobile app that enables them to enter the identifier corresponding to their Local Sandbox. By leveraging Signadot's operator, all requests from this session are routed to the version of the backend Service(s) running on the developer's workstation. This setup enables the locally running service to interact with the services running in the production environment, facilitating end-to-end testing.

With this workflow in place, developers now have the capability to thoroughly test specific user flows in the production environment from the front end. This approach gives them a high level of confidence in the functionality and stability of their code changes when deployed to production. By eliminating the need for a separate staging environment, this streamlined process enables developers to iterate and refine their work more efficiently, reducing potential bottlenecks and improving overall development speed.

“Our vision when we started talking to Signadot was that we wanted to have the same stack for testing and production. Signadot has allowed us to do just that, bringing down the lead time for our developers to test changes from 30 minutes to literally 60 seconds.”

-Amit Gud, Software Engineer, DoorDash

Results

The implementation of Signadot's workflow has significantly reduced the time required for high-fidelity testing from the front end during each iteration. Previously, developers had to wait for CI or deployments to the staging environment, which took around 30 minutes. Now, with the new approach, developers can complete their testing within a few minutes directly from the front end.

Developers benefit from a rapid iteration process on their local workstations. With Signadot, each change is quickly tested end-to-end against the production environment within a few minutes. This also incentivizes developers to engage in more ad-hoc testing, allowing them to validate their code changes effectively.

As a result, the number of rollbacks and production incidents has decreased, saving valuable developer time. With fewer disruptions caused by issues or bugs, developers can focus their efforts on building new features, enhancing productivity and enabling developers to make progress in delivering innovation to users.

Additionally, the elimination of the pre-production environment has significantly reduced cloud costs for DoorDash. Resources are allocated more efficiently, resulting in a more streamlined and cost-effective development process.

‍