TechTransitions

Implementing a Security-Centric Cloud Native CI/CD Pipeline: A Real-World Demonstration (using Terraform and GCP)

Implementing a Security-Centric Cloud Native CI/CD Pipeline: A Real-World Demonstration (using Terraform and GCP)

Chetan Thapliyal's photo
Chetan Thapliyal
·

10 min read

Table of contents

In today's software development landscape, the swift delivery of new features and updates is paramount; however, this must be balanced against the increasing importance of robust security practices. CI/CD (Continuous Integration/Continuous Delivery or Deployment) pipelines provide a framework for automating the building, testing, and deployment of applications, bolstering both speed and reliability. This project demonstrates a CI/CD pipeline where security is integrated as a first-class citizen throughout every stage.

Project Overview

The core objective of this project is to establish a CI/CD pipeline that prioritizes the following principles:

  • Security by Design: Security considerations are embedded in all phases of the development and deployment workflow.

  • Automation: Leveraging automation to maximize efficiency, reduce potential human error, and enforce security best practices.

  • Continuous Monitoring: Implementing systems and application-level monitoring for proactive issue detection and rapid response.

  • Infrastructure as Code with Terraform: Utilizing Terraform, a popular Infrastructure as Code (IaC) tool, to predictably create, change, and improve cloud infrastructure.

  • Google Cloud as the Cloud Platform: Leveraging Google Cloud's compute engine services and products to provision and manage the required infrastructure components.

Key Technologies

  • Kubernetes: Container orchestration for application deployment and management.

  • Jenkins: CI/CD automation server.

  • SonarQube: Static code analysis to ensure code quality and identify potential security issues.

  • Aqua Trivy: Vulnerability scanning for code dependencies and container images.

  • Nexus Repository: Secure storage for build artifacts.

  • Docker: Containerization for application packaging.

  • Docker Hub: Docker image registry.

  • Kubeaudit: Tool to audit Kubernetes clusters for various different security concerns.

  • Grafana: For system and application-level monitoring and alerting.

  • Prometheus: For collecting and querying metrics from services and endpoints.

  • Gmail: For status notifications and alerts.

Workflow

A client requests a feature or change in the application and creates a Jira ticket. The Jira ticket is then assigned to a developer.

  1. Development and Version Control:

    • Developers work on feature branches within a Git repository (e.g., GitHub) and successfully test the new feature in local environment.

    • Once changes are pushed along with source code to GitHub repository, it trigger the CI/CD pipeline.

  2. Build and Unit Testing:

    • The project's build system (e.g., Maven) compiles the code, and find out if there are any syntax base errors in the code.

    • Once code compilation is successful, Unit tests are executed to validate code functionality.

  3. Code Quality and Security Analysis:

    • SonarQube analyzes code for maintainability, potential bugs, and security vulnerabilities.

    • Aqua Trivy scans for vulnerabilities within project dependencies.

  4. Artifact Creation and Storage:

    • A build artifact (e.g., JAR, WAR) is generated.

    • The artifact is pushed to Nexus Repository for secure storage and proper release.

  5. Docker Image Creation:

    • Docker builds a container image incorporating the build artifact and tag it properly.

    • Aqua Trivy scans the image for vulnerabilities.

    • Docker image is pushed to dockerhub.

  6. Kubernetes Deployment:

    • Kube Audit secures the Kubernetes cluster.

    • The image is deployed to the Kubernetes cluster if all security scans pass.

  7. Mail Notification

    • Client and devops engineer will receive the mail notification if pipeline is successful or failed.

    • Notifications are sent to mail for deployment status, errors, and critical alerts.

  8. Monitoring:

    • Monitoring tools (Prometheus and Grafana) track the health of the system and application.

    • For system-level monitoring of hardware, Jenkins is used. Node Exporter monitors Jenkins.

Step-by-Step Project Execution

Phase 1: Creating Infrastructure

The project kicks off with the creation of an isolated network environment a critical step for maintaining security and control. This phase includes:

  1. Network Environment Creation: Setting up a dedicated and secure network environment to isolate and protect the project's infrastructure from external threats.

    • Create a Custom VPC on Google Cloud Platform with custom firewall rules allowing traffic to specific ports.

      Terraform Code:

        # Network creation
  #-----------------#
  resource "google_compute_network" "dev-cicd-vpc" {
      auto_create_subnetworks = true
      description             = "VPC for secure CICD pipeline."
      mtu                     = 1460
      name                    = "dev-cicd-vpc"
      project                 = var.gcp_project_id
      routing_mode            = "REGIONAL"
  }

  # Firewall Rules
  #-----------------#

  ## Custom firewall rules
  resource "google_compute_firewall" "dev-cicd-vpc-allow-custom" {
      name                    = "dev-cicd-vpc-allow-custom"
      project                 = var.gcp_project_id
      network                 = google_compute_network.dev-cicd-vpc.name
      description             = "Allows connection from any source to any instance on the network using custom protocols."
      direction               = "INGRESS"
      priority                = 65534
      source_ranges           = ["10.128.0.0/9", "0.0.0.0/0"] 
      allow {
          protocol = "tcp" 
          ports    = ["80", "443", "465", "6443", "3000-10000", "30000-32767"]
      }
  }

  ## ICMP
  resource "google_compute_firewall" "dev-cicd-vpc-allow-icmp" {
      network     = google_compute_network.dev-cicd-vpc.name
      project     = var.gcp_project_id
      direction   = "INGRESS"
      priority    = 65534
      source_ranges = ["0.0.0.0/0"]
      name        = "dev-cicd-vpc-allow-icmp" 
      description = "Allows ICMP connections from any source to any instance on the network."
      allow {
          protocol = "icmp"
      }
  }

  ## SSH
  resource "google_compute_firewall" "dev-cicd-vpc-allow-ssh" {
      network     = google_compute_network.dev-cicd-vpc.name
      project     = var.gcp_project_id
      direction   = "INGRESS"
      priority    = 65534
      source_ranges = ["0.0.0.0/0"]
      name        = "dev-cicd-vpc-allow-ssh"
      description = "Allows TCP connections from any source to any instance on the network using port 22."
      allow {
          protocol = "tcp" 
          ports    = ["22"]
      }
  }
💡
To install the various DevOps tools, I used Bash scripts and leveraged GCP's startup script feature when creating VMs.

Project Repo

  1. Kubernetes Cluster Deployment: Establishing a robust Kubernetes (K8s) cluster as the foundation for the project's deployment needs. This cluster is thoroughly scanned and secured to ensure a resilient platform.

    • Create 3 VM's on GCP for Kubernetes (1 Master and 2 Slave Nodes) and configure Kubernetes on them.

        ## Master Node VM
  resource "google_compute_instance" "cluster-instances-node-master" {
      boot_disk {
          auto_delete = true
          device_name = "k8-cluster-nodes"

          initialize_params {
          image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
          size  = 25
          type  = "pd-balanced"
          }

          mode = "READ_WRITE"
      }

      can_ip_forward      = false
      deletion_protection = false
      enable_display      = false

      labels = {
          goog-ec-src = "vm_add-tf"
          node        = "master"
      }

      machine_type = "e2-medium"

      metadata = {
          startup-script = file("./scripts/masterVM.sh")
      }

      name = "cluster-instances-node-master"

      network_interface {
          access_config {
          network_tier = "STANDARD"
          }

          queue_count = 0
          stack_type  = "IPV4_ONLY"
          network     = google_compute_network.dev-cicd-vpc.name
      }

      scheduling {
          automatic_restart   = true
          on_host_maintenance = "MIGRATE"
          preemptible         = false
          provisioning_model  = "STANDARD"
      }

      service_account {
          email  = var.gcp_service_account_email
          scopes = ["https://www.googleapis.com/auth/cloud-platform"]
      }

      shielded_instance_config {
          enable_integrity_monitoring = true
          enable_secure_boot          = false
          enable_vtpm                 = true
      }

      tags = ["http-server", "https-server", "lb-health-check"]
      zone = "asia-south1-c"
  }

  ## Slave Node VM (2 nodes)
  resource "google_compute_instance" "cluster-instances-node-slave" {
      count = 2

      boot_disk {
          auto_delete = true
          device_name = "k8-cluster-nodes"

          initialize_params {
          image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
          size  = 25
          type  = "pd-balanced"
          }

          mode = "READ_WRITE"
      }

      can_ip_forward      = false
      deletion_protection = false
      enable_display      = false

      labels = {
          goog-ec-src = "vm_add-tf"
          node        = "slave0${count.index + 1}"
      }

      machine_type = "e2-medium"

      metadata = {
          startup-script = file("./scripts/slaveVM.sh")
      }

      name = "cluster-instances-node-slave0${count.index + 1}"

      network_interface {
          access_config {
          network_tier = "STANDARD"
          }

          queue_count = 0
          stack_type  = "IPV4_ONLY"
          network     = google_compute_network.dev-cicd-vpc.name
      }

      scheduling {
          automatic_restart   = true
          on_host_maintenance = "MIGRATE"
          preemptible         = false
          provisioning_model  = "STANDARD"
      }

      service_account {
          email  = var.gcp_service_account_email
          scopes = ["https://www.googleapis.com/auth/cloud-platform"]
      }

      shielded_instance_config {
          enable_integrity_monitoring = true
          enable_secure_boot          = false
          enable_vtpm                 = true
      }

      tags = ["http-server", "https-server", "lb-health-check"]
      zone = "asia-south1-c"
  }

  2. Essential Server Setup: Configuring and deploying essential servers to support the CI/CD processes. This includes:

    • Jenkins Server: For continuous integration and deployment automation.

        # Jenkins VM
  #-----------------#

  resource "google_compute_instance" "jenkins" {
      boot_disk {
          auto_delete = true
          device_name = "jenkins-vm"

          initialize_params {
          image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
          size  = 30
          type  = "pd-balanced"
          }

          mode = "READ_WRITE"
      }

      can_ip_forward      = false
      deletion_protection = false
      enable_display      = false

      labels = {
          goog-ec-src = "vm_add-tf"
          jenkins     = ""
      }

      machine_type = "e2-standard-2"

      metadata = {
          startup-script = file("./scripts/jenkins.sh")
      }

      name = "jenkins"

      network_interface {
          access_config {
          network_tier = "STANDARD"
          }

          queue_count = 0
          stack_type  = "IPV4_ONLY"
          network     = google_compute_network.dev-cicd-vpc.name
      }

      scheduling {
          automatic_restart   = true
          on_host_maintenance = "MIGRATE"
          preemptible         = false
          provisioning_model  = "STANDARD"
      }

      service_account {
          email  = var.gcp_service_account_email
          scopes = ["https://www.googleapis.com/auth/cloud-platform"]
      }

      shielded_instance_config {
          enable_integrity_monitoring = true
          enable_secure_boot          = false
          enable_vtpm                 = true
      }

      tags = ["http-server", "https-server", "lb-health-check"]
      zone = "asia-south1-c"
  }

    • SonarQube Server: For continuous code quality and security analysis.

        # Sonarqube VM
  #---------------#
  resource "google_compute_instance" "sonarqube" {
      boot_disk {
          auto_delete = true
          device_name = "k8-cluster-nodes"

          initialize_params {
          image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
          size  = 20
          type  = "pd-balanced"
          }

          mode = "READ_WRITE"
      }

      can_ip_forward      = false
      deletion_protection = false
      enable_display      = false

      labels = {
          goog-ec-src = "vm_add-tf"
          sonarqube   = ""
      }

      machine_type = "e2-medium"

      metadata = {
          startup-script = file("./scripts/sonarqube.sh")
      }

      name = "sonarqube"

      network_interface {
          access_config {
          network_tier = "STANDARD"
          }

          queue_count = 0
          stack_type  = "IPV4_ONLY"
          network     = google_compute_network.dev-cicd-vpc.name
      }

      scheduling {
          automatic_restart   = true
          on_host_maintenance = "MIGRATE"
          preemptible         = false
          provisioning_model  = "STANDARD"
      }

      service_account {
          email  = var.gcp_service_account_email
          scopes = ["https://www.googleapis.com/auth/cloud-platform"]
      }

      shielded_instance_config {
          enable_integrity_monitoring = true
          enable_secure_boot          = false
          enable_vtpm                 = true
      }

      tags = ["http-server", "https-server", "lb-health-check"]
      zone = "asia-south1-c"
  }

    • Monitoring Servers: To support system and application-level monitoring from the outset.

        # Monitoring VM
  #-----------------#

  resource "google_compute_instance" "monitor" {
      boot_disk {
          auto_delete = true
          device_name = "monitoring"

          initialize_params {
          image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
          size  = 20
          type  = "pd-balanced"
          }

          mode = "READ_WRITE"
      }

      can_ip_forward      = false
      deletion_protection = false
      enable_display      = false

      labels = {
          goog-ec-src = "vm_add-tf"
          monitor     = ""
      }

      machine_type = "e2-standard-2"

      metadata = {
          startup-script = file("./scripts/monitoring.sh")
      }

      name = "monitor"

      network_interface {
          access_config {
          network_tier = "PREMIUM"
          }

          queue_count = 0
          stack_type  = "IPV4_ONLY"
          network     = google_compute_network.dev-cicd-vpc.name
      }

      scheduling {
          automatic_restart   = true
          on_host_maintenance = "MIGRATE"
          preemptible         = false
          provisioning_model  = "STANDARD"
      }

      service_account {
          email  = var.gcp_service_account_email
          scopes = ["https://www.googleapis.com/auth/cloud-platform"]
      }

      shielded_instance_config {
          enable_integrity_monitoring = true
          enable_secure_boot          = false
          enable_vtpm                 = true
      }

      tags = ["http-server", "https-server", "lb-health-check"]
      zone = "asia-south1-c"
  }

Phase 2: Version Control and Notification System

In this phase, the focus is on version control and establishing a notification system to keep stakeholders informed:

  1. Version Control with GitHub:

    • Private Repository Setup: Source code is pushed to a private GitHub repository to ensure security and privacy.

    • Branching Strategy: Implementing a branching strategy (e.g., feature branches, development, main) to manage code changes efficiently.

  2. Mail Notification System:

    • Configuration: Setting up an email notification system to alert stakeholders about the project's progress and any critical issues.

    • Integration: Integrating the notification system with the CI/CD pipeline to provide real-time updates on build statuses, deployments, and alerts.

Phase 3: Building the CI/CD Pipeline

The CI/CD pipeline is the heart of the project, automating the deployment process and ensuring robust security measures:

  1. Pipeline Design and Implementation:

    • Jenkins Pipeline Configuration: Defining a declarative Jenkins pipeline (Jenkinsfile) that outlines the steps for building, testing, and deploying the application.

  2. Security Measures:

    • Code Quality Analysis with SonarQube: Integrating SonarQube to perform static code analysis and ensure code quality.

    • Vulnerability Scanning with Aqua Trivy: Scanning code dependencies and Docker images for vulnerabilities to enhance security.

  3. Automated Deployment:

    • Artifact Creation and Storage: Building the application artifacts (e.g., JAR, WAR) and securely storing them in Nexus Repository.

    • Container Image Build and Scan: Creating Docker container images and scanning them for vulnerabilities before deployment.

    • Kubernetes Deployment: Automating the deployment of container images to the Kubernetes cluster, ensuring a smooth transition from development to production.

Phase 4: Monitoring at Scale

In the final phase, system-level and application-level monitoring are implemented to maintain the health of the application and provide real-time insights:

  1. System Monitoring with Prometheus and Grafana:

    • Metric Collection: Prometheus collects metrics from various services and endpoints, providing detailed insights into system performance.

    • Visualization and Alerting: Grafana visualizes these metrics and sets up alerting mechanisms to notify of any performance issues or anomalies.

  2. Application Monitoring:

    • Endpoint Monitoring with Blackbox Exporter: Ensuring that web endpoints are monitored for availability and responsiveness.

    • Traffic and Performance Monitoring: Monitoring website traffic and application performance to ensure optimal user experience.

  3. Continuous Improvement:

    • Feedback Loop: Using monitoring data to identify areas for improvement and optimize the CI/CD pipeline and deployment processes continually.

    • Proactive Maintenance: Implementing proactive measures based on monitoring insights to prevent issues before they impact users.

Security Considerations

  • Pre-deployment Kubernetes Security: The Kubernetes cluster is hardened to minimize potential attack surfaces.

  • Pipeline as Code: Pipeline configuration (e.g., Jenkinsfile) is treated as code, enabling versioning and peer review.

  • Vulnerability Scanning: Scans at the code and image stages proactively identify issues.

  • Monitoring: Anomalies or suspicious behavior are detected early.

Benefits of a Secure CI/CD Pipeline

  • Enhanced Security Posture: Integration of security measures throughout the process mitigates risks.

  • Increased Confidence: Rigorous testing and security checks lead to more reliable releases.

  • Improved Agility: Automation streamlines development and reduces time to market.

  • Reduced Operational Overhead: Fewer manual processes minimize potential errors.

Conclusion

This project exemplifies a comprehensive CI/CD pipeline that can serve as a blueprint for organizations looking to implement or refine their own CI/CD strategies. It demonstrates the importance of each phase and the inter-connectedness of various tools and practices that contribute to a successful software delivery lifecycle.

Did you find this article valuable?

Support Chetan Thapliyal by becoming a sponsor. Any amount is appreciated!

GCPci-cdDevopsTerraformprojects