DevOps tooling as code

Every DevOps team has a Jenkins server that someone set up by hand years ago and everyone is afraid to touch. I wanted ours to be something we could rebuild on a Friday afternoon without worrying.

The setup

• Jenkins and Nexus in Docker. Both ran as containers, so upgrades meant changing an image tag instead of logging into a server.
• Terraform for the infrastructure. The machines, networks and security groups on AWS were all defined in Terraform. If something broke, we could recreate it.
• Ansible and Chef for configuration. Application settings and server configuration lived in code, not in someone's memory.
• GitLab and Bitbucket for everything. Pipelines, Terraform and playbooks were all versioned and reviewed like application code.

Fixing Java pipelines

A lot of my day-to-day was fixing CI/CD pipelines for Java applications. The failures were rarely exotic. Most came from three places:

1. Dependencies that changed underneath us. Pinning versions and caching them in Nexus fixed most of these.
2. Builds that depended on the machine they ran on. Building inside containers made them repeatable.
3. Tests that only failed sometimes. These need fixing, not retrying.

What I learned

• If you can't rebuild your tools from code, you don't really own them.
• Small, readable pipelines beat clever ones. The next person to debug it might be you, at 2am.
• Infrastructure as code is also documentation. New team members learned the setup by reading the repo.