Cypress on Heroku

by Bryan Li, Software Engineer @ Resilia

Here at Resilia, we have fully embraced the 12 Factor App Principle. One of the key components to increasing our confidence while maintaining our development velocity is to have an appropriate level of integration test coverage. These are some of the technical hurdles we faced and decisions we made when increasing our coverage.

Pathway to CI/CD: Integration Testing With Cypress

Cypress is one of the coolest integration testing tools in the JavaScript ecosystem. We are leveraging Cypress to build towards a fully automated CI/CD pipeline where engineers are confident in each production promotion.

While it would be great to explain the benefits of having integration testing and the various benefits of Cypress, this article will focus on the challenges of running Cypress within our existing infrastructure.

Before we talk about the challenges, it is important to have some context of our CI/CD pipeline and infrastructure to understand the reasons behind some of our decisions. We have a Github integration with Heroku CI where unit tests are executed on each PR. We have two static environments that represent the `main` branch: Staging and Production. We had spent a considerable amount of effort to match the two environments to allow us to perform extensive manual testing before production promotion. Lastly, we say production promotion because our process is moving built artifacts from one environment to another.

Challenge 1: Deciding Where Should Run Cypress

Once we decided to move forward with Cypress being our integration testing tool, we were faced with the choice of where to invoke Cypress. A very common choice is to run Cypress on the PRs, and we will have confidence in catching bugs that could be introduced to the Staging environment.

However, since we are already running unit tests on PRs, we wondered if we could make a case to move the integration tests to run on the actual Staging environment. There are several benefits of moving the integration tests behind the Staging environment. The first one is less time it takes to merge to main because we do not have to wait for the build and integration to run. The second is less dyno usage because theoretically the test would run on open PRs and all subsequent commits. Third, we do not have to provision resources in a test environment. Lastly, since the Staging environment matches Production very closely, we can rely on Cypress to mimic the actual user’s experience when using our app.

At the end, we decided to run Cypress on our Staging environment each evening. This limits us on deploying in the morning, which is something we are working on eliminating by having it run after each Staging deploy.

Challenge 2: Persisting Authenticated Session Between Tests

The second challenge is to persist user sessions between tests. As best practice, Cypress clears up localStorage and cookies between tests. It is great because you know each test is run independently. However, it is very difficult when you want to test features using an authenticated user session.

The common practice is to stub token or authentication credentials because of the mantra “Don’t test things you cannot control”. It is a great rule of thumb but it poses some challenges depending on your authentication service provider.

At Resilia, we use the latest industry best practices to secure our app. Our authentication flow does not handle passwords, and our application does not allow for passwords to exchange for Access Token. To dive deeper into different authentication flows, Auth0 has a great article here. Bottom line, there is no way within our application where we can stub a user session unless we explicitly allow it on an app-wide level.

To overcome the challenge, we group all of our tests within an authenticated session under one context block. Before we invoke the assertions, we invoke a custom login command where we input the test user’s credentials on a page hosted by our authentication service provider, and we persist localStorage and cookies and restore them before each assertion.

This approach may raise some eyebrows because we are “testing” functionality of a third party software. With that said, the login flow is part of our normal app experience, we have control over the login redirect, and we do have control over the login page (we can embed custom JS code and custom CSS on the template). They are arguments that justify incorporating the interactivity of the login page within our tests.

There are many issues with this approach. One issue we are solving right now is Cypress does not work on cross-domain websites. It is a pain for local development for third party hosted login page because tests running on `localhost` do not allow us to interact with the login page at `https://login-dev.authentication.com` to create a session.

Challenge 3: Running Cypress Within Heroku

The final challenge is to run Cypress on Heroku. Our goal is to run headless tests, and upload the result to cypress.io. However, the biggest problem is Heroku standard stacks run in a bare-bone Linux environment, and it requires xvfb (remember this as it will haunt you) package to run Electron, which is the bundled environment for headless tests.

Our first approach was to find official buildpacks that would support Cypress. On the official front, Cypress officially ended support for Heroku stacks. Majority of the existing solutions that we could find target outdated Heroku stacks. (UPDATE: this PR was merged 4 days prior to this article, and it stops spawning xvfb during headless mode. This means Cypress headless mode could be natively supported.)

Our second attempt was to use the existing Node buildpack. As expected, many dependencies are missing. Upon research, we concluded that maybe we can use the Google Chrome buildpack and run headless Cypress tests with chrome. We were still running into the same dependency errors and we opted to move forward with our next implementation. There are some funny bits like this.

Our third attempt was to create our scripts to install all the necessary dependencies through `apt-get`. However, the ending slug size exceeded the Heroku slug size limit of 500mb.

The solution was Docker. Cypress provides an official Docker image that has all the necessary dependencies. In addition, the best part is Heroku has no limit on the size of Docker containers. You can follow the steps to build your own docker container in Heroku here.

Finally, our Cypress application is deployed, and we can use Heroku Scheduler to schedule a nightly integration test that’s hooked up with Slack, where we can quickly check on the status of the tests before we promote in the morning.

Final Thoughts

With the current setup, we have begun our journey to the fabled land of CI/CD. However, there is still more work to be done. The first step is to automate integration tests after each staging deployment to allow us to deploy more often. The second is to set up a local development authentication utility to allow us to stub the user session to allow developers to write tests more efficiently.