Avoiding Bit Rot in Your Build Pipeline

MAY 17, 2020


You go to make a quick fix on a service that’s been stable and otherwise unchanged for months. It takes you minutes to complete. However, when you go to deploy the fix, your super fancy Continuous Integration Build and Deployment (CI/CD) Pipeline that’s supposed to compile and push your code out fails. And it fails hard blocking you from building and deploying your code in a production-ready manner.

You spend a few hours debugging the failing pipeline and eventually discover that a dependency of a dependency of a dependency of a dependency1 updated its API in a breaking way, and despite the package stating it uses Semantic Versioning2 only the patch-level version of the dependency was bumped by the author. So now your code is exploding since the build pipeline dutifully pulled in the updated code thinking it was a non-breaking change.

Suggested Mitigations

In order to avoid this scenario, I recommend:

  1. Pin your dependencies3 by using a lockfile for your dependency tool and specificying the tightest version constraints you can get away with wherever versions are used (e.g. in the FROM image:version directive of a Dockerfile) , and
  2. Schedule your build pipeline to run at least as frequently as every night so you’ll notice failures proactively instead of when you need to deploy an emergency patch.

Even if you think your build is deterministic, there’s most likely some level of non-determinism in there. While you might have a lockfile checked in, maybe you forgot to pin the Docker container version4.

If you’re using GitHub Actions, you can use the on.schedule directive.

If you’re using GitLab CI, you can use its Scheduled Pipelines feature.

By scheduling a run of your build pipeline, you will ensure your CI/CD pipeline is exercised even if commits in that project slow down and that it remains ready for patches.

Hopefully this tip will help you avoid a headache in the future.

  1. Could you tell I was referring to a Node.js project? Node modules have notorious dependency graphs: https://www.reddit.com/r/ProgrammerHumor/comments/6s0wov/heaviest_objects_in_the_universe/.

  2. From https://semver.org/:

    Given a version number MAJOR.MINOR.PATCH, increment the:

    1. MAJOR version when you make incompatible API changes,
    2. MINOR version when you add functionality in a backwards compatible manner, and
    3. PATCH version when you make backwards compatible bug fixes.

    Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

  3. If you pin your dependendencies, be sure to add automation in your CI/CD pipeline to check for security updates. With Node modules, this may be as easy as running npm audit --production in your build pipeline, and then failing the pipeline if that command fails so someone can manually intervene. For Docker images, if you are using AWS’ Elastic Container Registry (ECR) solution, enable image scanning and alert on the scan results.

    If you don’t want to think about scanning for dependencies, checkout https://snyk.io/ which can handle all sorts of scanning and patching for you!

  4. This has happened to me. I went to get a project up and running quickly, and used FROM ubuntu:latest in my Dockerfile. Everything was fineā€¦until months later when the latest pointed to a major update of Ubuntu compared to the last time the project was changed and built.