A infrastructure pipeline is an automated process for building, testing, and deploying your IT infrastructure. The developer write a code to define the infrastructure, and the pipeline automate the rest of the steps.

At the heart of this is CI/CD:

  • Continuous Integration (CI): This is the practice of frequently merging your code changes into a central repository. Each time you do, an automated build and test are run.
  • Continuous Delivery/Deployment (CD): This is the practice of automatically deploying your code to a testing or production environment after it has passed the CI stage. MartinF

CI/CD tools can be used to test server configuration (e.g., Chef cookbooks, Puppet modules, Ansible playbooks), server image building (e.g., Packer), environment provisioning (e.g., Terraform, CloudFormation) and integration of environments. The use of pipelines for infrastructure as code enables errors to be found before changes are applied to operational environments. They also offer a way to ensure that infrastructure tooling is run consistently, from CI/CD agents TW - InfraPipeline

Benefits of automation:

  • Keeps things reproducible
  • Avoid time spent on documenting manual steps
  • Provides reliable processes
  • Teams focus on the actual change rather than the steps to apply them
  • Improves traceability of changes

Pipeline:

  • Reliable: a reproducible process to provision our infrastructure/application
  • Fast: quick feedback cycles if the pipeline succeeds or fails
  • Specific: concrete feedback on what went wrong when errors happen

Pipeline for Infrastructure or application has similar benefits and they can be created to run together or in separate pipelines. It will depends on the strategy (small stacks, run in parallel, etc). [Tw - InfraPipeline2]

  • Put all of the environments into a single stack (bad)
  • Define each environment in a separate stack (works well for simple setups)
  • Create a single stack definition and promote it through a pipeline (works well for larger and more complex groups)


Stages of an Infrastructure Pipeline

  • Source: write the code and push to a version control
  • Build: first validation and package (if apply). This might involve checking for syntax errors and preparing it for deployment.
  • Test: Before you deploy your infrastructure, you need to make sure it’s going to work as expected. These tests can happen in different moment. It depends the team decisions based on the time needed for each one. Some of them can be done in PR to avoid impact the pipeline performance.
  • Deploy: Once your code has passed all the tests, it’s time to deploy it to your target environment.
  • Monitor: After your infrastructure is up and running, you’ll want to monitor its performance and health to make sure everything is running smoothly.


Testing the Infrastructure

The best place to perform most of your testing is in the Test stage after the Build stage and before the Deploy stage. In terms of application it is clear. However, when we talk about infrastructure tests the answer is not so precise.

Infrastructure test can involve offline tests, that run in a virtual environment (local container, using mocks), or online tests that interact with cloud infrastructure. The tests can spend long time and impact the pipeline's performance. To avoid that, they could be executed in different moments and even use a isolated environment.

Types of Infrastructure Tests:

  • Static Analysis: check the code for syntax errors, security vulnerabilities, and compliance with best practices without actually running it. Tools like TFLint, TFsec, and Checkov are great for this
  • Unit Testing: test individual components of the infrastructure in isolation
  • Integration Testing: test how different components of the infrastructure work together
  • End-to-End Testing: test the entire infrastructure from start to finish
  • Compliance Testing: check to ensure the infrastructure complies with all relevant security and regulatory standards. Tools like InSpec are often used for this.


Infrastructure as Code (IaC)

Infrastructure as Code (IaC) is the practice of managing and provisioning your infrastructure through code instead of through manual processes, which makes it easier to edit and distribute configurations.

Popular IaC tools include:

  • Terraform: An open-source tool that lets you define and provision infrastructure across a variety of cloud providers.
  • Ansible: An open-source tool that automates software provisioning, configuration management, and application deployment.
  • AWS CloudFormation: A service that helps you model and set up your Amazon Web Services resources.

IaC is the engine that drives the infrastructure pipeline. It allows you to automate the entire process of creating, testing, and deploying your infrastructure.

Stages:

  • Code (ΩDeclarative description for Terraform) -> Describe cloud resources
  • Push to Git Repo
  • Trigger PIPELINE
  • Pipeline: build -> Test -> Deploy

Challanges:

  • If stacks are split, they may still have dependencies
  • Orchestrating changes between stacks can get messy
  • How to avoid tight coupling
  • Support re-use and testing of stacks

Advices:

  • use mocks for isolated testing
  • integrating with tools data structures
  • minimize variability: is you need a module to do two or more different things, maybe you need two or more modules
  • One concern per module: the purpose of each module should be clear, with limited scope
  • Don’t fear duplication: reuse code with the same purpose
  • Running TF in pipeline
  • Multiple pipelines if necessary
  • Optimize for MTTR
  • Go for immutable infra if possible
  • Locking stateful resources (lock_level = “CanNotDelete”)
  • Modularizing TF Code -> Allows test smallest piece of code
  • Splitting state
  • Backup and Recovery
  • Least Privileges for Pipeline

Considerations when running Terraform in Pipeline:

  • disable all manual input (-input=false)
  • archive Terraform plan (terraform plan -out=tfplan ; terraform apply tfplan)
  • Package .terraform folder and restore at the same absolute path
  • turn on auto-approve
  • Each stack in its own pipeline


Challenges in Infrastructure Automation

  • Learning Curve: Adopting IaC requires new skills and a new way of thinking.

  • Configuration Drift: This happens when the actual state of your infrastructure no longer matches the code that defines it. This can happen when people make manual changes to the infrastructure.
  • Complexity: As your infrastructure grows, so does the complexity of your IaC code. It can become difficult to manage and maintain. Mainly if it involves refactor.
  • Security: Storing sensitive information like passwords and API keys in your code is a major security risk. You’ll need to use a secrets management tool like HashiCorp Vault or AWS Secrets Manager.


Solving Conventional Problems

  • Slow Deployments: Manual infrastructure provisioning is slow and error-prone.

  • Inconsistency: When you have multiple people setting up infrastructure manually, you’re bound to end up with inconsistencies.

  • Lack of Visibility: It can be difficult to know what’s going on with your infrastructure when it’s all being managed manually.

  • Security Vulnerabilities: Manual processes can lead to security holes.


Infrastructure vs. Application Pipelines

The both pipelines share many of the same principles. However, there are some key differences:

Purpose:

  • Infrastructure Pipeline: To build, test, and deploy infrastructure.
  • Application Pipeline: To build, test, and deploy applications.

Artifact:

  • Infrastructure Pipeline: Infrastructure code (e.g., Terraform, Ansible).
  • Application Pipeline: Application code (e.g., Java, Python, Go).

Testing

  • Infrastructure Pipeline: Focuses on infrastructure health and compliance.
  • Application Pipeline: Focuses on application functionality and performance.


DevSecoOps

DevSecOps [1][2][3][4] is a cultural and technical shift that integrates security practices into every stage of the DevOps lifecycle. The goal is treat infrastructure's security like you treat the application's security: as an integral part of the code itself .

Add this in stages:


Summary