Testing the changes in a pull request isn't intuitive: checking out the branch, waiting for Docker to launch all service containers, keep waiting for another bit, then getting to test the actual PR. By the time you reach the last step you might have forgotten what you initially wanted to do, or another distraction came up in between. Does it have to be that way?
With remote development environments like GitHub Codespaces, pulling up development environments on-demand has gotten substantially more accessible, but the problems are still the same, as you have to start everything up manually. And even then, running services on the same instance usually doesn't reflect your production infrastructure.
We clearly need a different approach that is as close to production as physically possible, that can be started up and torn down for every pull request, that helps us to test changes when we need to.
For the frontend, Vercel offers preview deployments on every pull request and whenever else you need it. Creating a preview is as easy as setting up Vercel for GitHub or running vercel. But since most teams usually deploy their backend separately, we need more to cover the entire stack from end to end.
In this guide, we'll walk through some planning steps, laying out what we'd ideally want, and then get there step by step. In the end, you'll be able to spin up full-stack preview environments for every pull request, making use of powerful tools like Pulumi, GitHub Actions, and Vercel.
Planning Preview Environments
With Vercel, previewing frontend changes is incredibly easy. Just install one of their Git integrations, open a PR, and you're all set: Vercel will spin up preview deployments for every PR and keep them updated whenever you push further changes.
This solves the frontend part, but what if we changed something in the backend? If we're working in a monorepo setting, we might create a PR that contains changes across the stack, and testing that would require spinning up all resources needed to run our backend.
Too Many Services
Imagine you're using third-party services like AWS, Cloudflare DNS, NewRelic for monitoring, or other infrastructure providers. While we could script that work away, setting everything up manually not only becomes increasingly complex but also prone to errors when you're dealing with more than a simple setup, and we'd probably spend a lot of time building abstractions other people have already built and fixed over time.
By now, you might have noticed the theme going on here: It sounds like we need a healthy dose of automation to spin up an environment that mimics production and update itself on changes until we decide to tear it down again. Enter infrastructure-as-code with Pulumi.
Pulumi to the rescue
At Hygraph, we've been using Pulumi for more than a year now, and I'm almost confident that I won't accidentally delete production. Jokes aside, Pulumi is a serious competitor to Terraform because it reframes the approach of IaC tooling to use languages your team is already using, including TypeScript, Go, Python, and .NET. It integrates with more than fifty cloud providers, including the big public clouds, and other providers out there using existing Terraform providers, giving you access to databases, monitoring, networking, and version control, so your stack is fully covered.
IaC works best when you fully adopt it. When you rely on manual work to set up some resources, that will bite you when you need to spin up environments at any time. So if you really want a backend-on-demand, you need to automate every last bit of what you're deploying. You might think that it's impossible to account for steps like setting up a database or copying some files to a remote host, but with dynamic providers, pretty much any case can be supported. Since dynamic providers are only supported in Node.js and Python right now, I'll create my Pulumi project with TypeScript in this post.
What we'll need
In this guide, we'll use GitHub Actions to integrate tightly with Pull Requests. In theory, this approach should work on any platform that has similar capabilities. Here's what should happen:
- We work on some changes and open a pull request
- If this pull request contains frontend changes, Vercel will create a new preview deployment
- To spin up a full backend preview environment, we attach a specific label, let's call it
preview-backendfor the rest of this guide - Once the preview deployments are ready, we'll get a comment including links with instructions to check those out
- When we're done previewing, we can tear down the backend preview again
This flow is simple enough for everyone in the team to follow, as it has as little friction as possible. You always get a preview for frontend changes because that's a common place to give feedback on design and functionality. Whenever you want to see the backend changes in action, just label the PR and CI will take care of it. When you're done, just remove the label, and you're done.
Now let's make it happen!
Getting there
Frontend Previews for every pull request
The first step is easy enough, follow the Vercel documentation to set up GitHub as the Git provider of choice. Once you've authorized the Vercel GitHub app, every new commit pushed will get a preview deployment. If that's too much for your taste, you can also use the CLI to deploy previews only when you need them, for example only if files related to the frontend changed.
Plugging in any backend to our frontend
The second step is to extend your frontend to allow a custom API endpoint. This is necessary so that you can switch to any backend of choice, which you can use for backend previews later on. This step is optional if you want to preview the backend in isolation (i.e. without frontend).
You could allow to hook up a custom backend in multiple ways, an easy one would be to set a default and allow overrides via a custom route or local storage.
Setting this up is quite specific to your stack and environment, but the concept should be simple, just allow connecting to a different backend, and if this will happen a lot, make sure it's easy and understandable.
You might want to disable or restrict that logic for production builds. This could be done by checking for an environment variable included only in preview or production builds.
Backend previews on demand
With our frontend in place, we can focus on the backend. Our GitHub action will have two main entry points: Labeling, pushing a change if the PR is labeled, and un-labeling. We need all of those to spin up a preview environment, redeploy whenever a change is pushed, and tear it down again once we're done.
The faster we can spin up our backend, the better. Some resources unfortunately take quite some time to be created (looking at you, ELB and RDS for PostgreSQL), but most will be up quickly. We'll probably only create most resources once anyway since updates to a PR are just code changes, which cause a redeploy of some containers, but not the whole infrastructure.
The more services you use, the more expensive it will be to run multiple additional environments, so creating those on demand is great.
Preview Deployments with Pulumi
To spin up backend previews with Pulumi, our project needs to contain an existing production deployment, that should be used as a template. This is important because we need some state we can replicate for our preview, including service and deployment settings.
How many configuration options you expose heavily depends on your setup, so whether you expose a lot of different configuration options per stack or create equal stacks by using the same code is completely up to you.
In our case, we just want to get a preview that is as similar to production as possible, but you could create instances with fewer compute resources for preview environments, for example.
To make this all work, we'll use two important options in every stack configuration: preview and commit. preview is a boolean flag that will toggle preview-specific resources to be created. commit is the Git commit SHA that our services should run. Updating this will lead to our service deployment being updated as well.
import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';
import * as awsx from '@pulumi/awsx';
import * as cloudflare from '@pulumi/cloudflare';
const stackName = pulumi.getStack();
const config = new pulumi.Config();
// Load configuration items, commit and zoneId are required
const commit = config.require('commit');
const zoneId = config.require('zoneId');
const preview = config.boolean('preview') === true;
// TODO Configure registry
const apiImage = `<REGISTRY>/api:${commit}`;
// Create load balancer
const lb = new awsx.lb.ApplicationLoadBalancer('main', {});
// Forward DNS traffic from Cloudflare to Load Balancer
new cloudflare.Record('to-lb', {
name: preview ? `preview-${stackName}` : `api`,
type: 'CNAME',
value: lb.loadBalancer.dnsName,
proxied: false,
ttl: 60,
zoneId
});
// Connect to container at port 80
const appListener = new awsx.lb.ApplicationListener('forward-to-instance', {
port: 80,
protocol: 'HTTP',
loadBalancer: lb
});
// Finally, create ECS Fargate service for our API
new awsx.ecs.FargateService('api', {
taskDefinitionArgs: {
containers: {
api: {
image: apiImage,
memory: 128,
portMappings: [appListener]
}
}
},
// Spin up two instances
desiredCount: 2
});
This example configuration will deploy a DNS record on Cloudflare that uses the Elastic Application Load Balancer deployed on AWS. In production, the URL is api.<zone>, in preview it's preview-<stack>.zone. Requests are routed through the Listener to one of two deployed API instances. Whenever the commit is updated, the instances will update as well.
This setup is simplified a lot for brevity, in a production setup, you would want to switch to TLS traffic and probably add more services.
Automating with GitHub Actions
Let's get started with our gitHub Actions workflow. In a new YAML file in the .github/workflows directory, we'll add our workflow definition
name: Backend Preview
on:
pull_request:
types:
- labeled
- unlabeled
- synchronize
concurrency: preview_${{ github.event.number }}
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
jobs:
This workflow will run when labels are added or removed, or a new commit is pushed to the PR (or the branch is updated otherwise). The concurrency setting makes sure we only run this workflow at most once at a given time for the same PR, preventing any conflicts from parallel access. We also pass a Pulumi access token through the environment.
With this added, let's cover all three workflow parts.
When adding the preview-backend label
The first event we're interested in is when the preview-backend label is added to our PR. This condition is realized with the if: option on the job-level. We'll go on to check out the PR branch, create a new stack, copy over configuration values from the current production stack, and push those changes.
In this step, we do not actually create any cloud provider resources yet. Instead, we handle stack creation just once, in the update case.
create-backend-preview:
runs-on: ubuntu-latest
if: ${{ github.event.action == 'labeled' && github.event.label.name == 'preview-backend' }}
steps:
# Check out PR branch
- uses: actions/checkout@v2
with:
ref: ${{ github.head_ref }}
- name: Install Pulumi CLI
uses: pulumi/setup-pulumi@v2
- name: Create new stack
run: pulumi stack init --non-interactive --logtostderr "pr${{ github.event.number }}"
- name: Seed config
# Copy over Pulumi production config and toggle preview flag
# If the config is different across stacks in more ways, this is a place that would require refactoring
run: |
pulumi config cp --non-interactive --logtostderr --stack production --dest "pr${{ github.event.number }}"
pulumi config set --stack "pr${{ github.event.number }}" preview true
# Commit stack file
- name: Push changes
run: |
git config user.name github-actions
git config user.email github-actions@github.com
git add .
git commit --message "Create backend preview stack pr${{ github.event.number }}"
git push
When pushing a commit with the preview-backend label added
Whenever we detect a change in the PR branch while the label is present (and thus while the preview environment exists, we will prepare all services with the latest code (not shown here as it depends on your environment) and apply the changes in Pulumi.
If we haven't run pulumi up previously, this step will initialize all resources.
update-backend-preview:
if: ${{ github.event.action == 'synchronize' && contains(github.event.pull_request.labels.*.name, 'preview-backend') }}
# Example: Run this after all services are built
# needs:
# - build-services
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Install Pulumi CLI
uses: pulumi/setup-pulumi@v2
- name: Update commit
run: pulumi config set --stack "pr${{ github.event.number }}" commit ${{ github.event.after }}
# Apply changes (pull new image, update containers, etc.)
- uses: pulumi/actions@v3
with:
command: up
stack-name: pr${{ github.event.number }}
Running this will deploy the latest commit on your infrastructure, fully automated.
When removing the preview-backend label
The last workflow step is to tear down our backend preview once the label is removed.
destroy-backend-preview:
runs-on: ubuntu-latest
if: ${{ (github.event.action == 'unlabeled' && github.event.label.name == 'preview-backend') }}
steps:
- uses: actions/checkout@v2
with:
ref: ${{ github.head_ref }}
- name: Install Pulumi CLI
uses: pulumi/setup-pulumi@v2
- uses: pulumi/actions@v3
with:
command: destroy
stack-name: pr${{ github.event.number }}
- name: Delete stack
run: pulumi stack rm --non-interactive --yes --logtostderr "pr${{ github.event.number }}"
- name: Push changes
run: |
git config user.name github-actions
git config user.email github-actions@github.com
git add .
git commit --message "Remove backend preview stack pr${{ github.event.number }}"
git push
We'll simply destroy the preview stack and delete it afterward, which will also remove the stack configuration file from the disk. We'll propagate those changes to our pull request branch by creating another commit.
Considerations for the future
This setup gets you quite far already, but there's a couple of improvements that can make it even better.
Automatic Backend Teardown
We went over the fact that we have to tear down the backend because we don't want to leave it running and consuming compute resources and money forever. If we stick with Pulumi, we have to destroy the stack in the same PR, to get rid of the stack configuration file again. You could of course just merge it in and have a scheduled action drop existing preview-related stacks periodically, but you cannot modify the same PR after merging it.
If some PRs are open for an extended period, you might want to shut down your previews after some time, keeping previews up only for up to 2 hours, for example.
Disallow merging in PR with preview stack additions
If you haven't set up a periodic cleanup action that will tear down preview stacks that got merged in, you could also add an action that checks if any preview stacks were added in a PR, and fails until those stacks are destroyed again. Similar to failing tests or a linter, this would forcefully remind you of tearing down the preview before you merge.
Commenting preview environment URLs
Once the frontend preview is deployed, Vercel will comment on your pull request, adding a link to it. If we want to, we can post details about the backend preview by exporting outputs from the stack.
// Expose API URL as Pulumi Output
export const apiUrl = `api-${stackName}.preview.example.com`;
After running pulumi up in the update case, you could add the following workflow steps
# Comment API deployment URL using stack output
- id: api-url
run: echo ::set-output name=apiUrl::$(pulumi stack output apiUrl --stack "pr${{ github.event.number }}")
- uses: actions/github-script@v5
with:
script: |
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: '${{ github.event.after }}: The backend preview is deployed at ${{ steps.api-url.outputs.apiUrl }} :rocket:'
})
This uses the GitHub Script action to create a comment with GitHub's REST API. By default, the user is the GitHub Actions application that is installed automatically by GitHub Actions. This way, every workflow run receives the GITHUB_TOKEN secret, also available in the github.token context.