Implementing a Security-Centric Cloud Native CI/CD Pipeline: A Real-World Demonstration (using Terraform and GCP)

Implementing a Security-Centric Cloud Native CI/CD Pipeline: A Real-World Demonstration (using Terraform and GCP)

ยท

10 min read

In today's software development landscape, the swift delivery of new features and updates is paramount; however, this must be balanced against the increasing importance of robust security practices. CI/CD (Continuous Integration/Continuous Delivery or Deployment) pipelines provide a framework for automating the building, testing, and deployment of applications, bolstering both speed and reliability. This project demonstrates a CI/CD pipeline where security is integrated as a first-class citizen throughout every stage.

Project Overview

The core objective of this project is to establish a CI/CD pipeline that prioritizes the following principles:

  • Security by Design: Security considerations are embedded in all phases of the development and deployment workflow.

  • Automation: Leveraging automation to maximize efficiency, reduce potential human error, and enforce security best practices.

  • Continuous Monitoring: Implementing systems and application-level monitoring for proactive issue detection and rapid response.

  • Infrastructure as Code with Terraform: Utilizing Terraform, a popular Infrastructure as Code (IaC) tool, to predictably create, change, and improve cloud infrastructure.

  • Google Cloud as the Cloud Platform: Leveraging Google Cloud's compute engine services and products to provision and manage the required infrastructure components.

Key Technologies

  • Kubernetes: Container orchestration for application deployment and management.

  • Jenkins: CI/CD automation server.

  • SonarQube: Static code analysis to ensure code quality and identify potential security issues.

  • Aqua Trivy: Vulnerability scanning for code dependencies and container images.

  • Nexus Repository: Secure storage for build artifacts.

  • Docker: Containerization for application packaging.

  • Docker Hub: Docker image registry.

  • Kubeaudit: Tool to audit Kubernetes clusters for various different security concerns.

  • Grafana: For system and application-level monitoring and alerting.

  • Prometheus: For collecting and querying metrics from services and endpoints.

  • Gmail: For status notifications and alerts.

Workflow

A client requests a feature or change in the application and creates a Jira ticket. The Jira ticket is then assigned to a developer.

  1. Development and Version Control:

    • Developers work on feature branches within a Git repository (e.g., GitHub) and successfully test the new feature in local environment.

    • Once changes are pushed along with source code to GitHub repository, it trigger the CI/CD pipeline.

  2. Build and Unit Testing:

    • The project's build system (e.g., Maven) compiles the code, and find out if there are any syntax base errors in the code.

    • Once code compilation is successful, Unit tests are executed to validate code functionality.

  3. Code Quality and Security Analysis:

    • SonarQube analyzes code for maintainability, potential bugs, and security vulnerabilities.

    • Aqua Trivy scans for vulnerabilities within project dependencies.

  4. Artifact Creation and Storage:

    • A build artifact (e.g., JAR, WAR) is generated.

    • The artifact is pushed to Nexus Repository for secure storage and proper release.

  5. Docker Image Creation:

    • Docker builds a container image incorporating the build artifact and tag it properly.

    • Aqua Trivy scans the image for vulnerabilities.

    • Docker image is pushed to dockerhub.

  6. Kubernetes Deployment:

    • Kube Audit secures the Kubernetes cluster.

    • The image is deployed to the Kubernetes cluster if all security scans pass.

  7. Mail Notification

    • Client and devops engineer will receive the mail notification if pipeline is successful or failed.

    • Notifications are sent to mail for deployment status, errors, and critical alerts.

  8. Monitoring:

    • Monitoring tools (Prometheus and Grafana) track the health of the system and application.

    • For system-level monitoring of hardware, Jenkins is used. Node Exporter monitors Jenkins.

Step-by-Step Project Execution

Phase 1: Creating Infrastructure

The project kicks off with the creation of an isolated network environment a critical step for maintaining security and control. This phase includes:

  1. Network Environment Creation: Setting up a dedicated and secure network environment to isolate and protect the project's infrastructure from external threats.

    • Create a Custom VPC on Google Cloud Platform with custom firewall rules allowing traffic to specific ports.

      Terraform Code:

        # Network creation
        #-----------------#
        resource "google_compute_network" "dev-cicd-vpc" {
            auto_create_subnetworks = true
            description             = "VPC for secure CICD pipeline."
            mtu                     = 1460
            name                    = "dev-cicd-vpc"
            project                 = var.gcp_project_id
            routing_mode            = "REGIONAL"
        }
      
        # Firewall Rules
        #-----------------#
      
        ## Custom firewall rules
        resource "google_compute_firewall" "dev-cicd-vpc-allow-custom" {
            name                    = "dev-cicd-vpc-allow-custom"
            project                 = var.gcp_project_id
            network                 = google_compute_network.dev-cicd-vpc.name
            description             = "Allows connection from any source to any instance on the network using custom protocols."
            direction               = "INGRESS"
            priority                = 65534
            source_ranges           = ["10.128.0.0/9", "0.0.0.0/0"] 
            allow {
                protocol = "tcp" 
                ports    = ["80", "443", "465", "6443", "3000-10000", "30000-32767"]
            }
        }
      
        ## ICMP
        resource "google_compute_firewall" "dev-cicd-vpc-allow-icmp" {
            network     = google_compute_network.dev-cicd-vpc.name
            project     = var.gcp_project_id
            direction   = "INGRESS"
            priority    = 65534
            source_ranges = ["0.0.0.0/0"]
            name        = "dev-cicd-vpc-allow-icmp" 
            description = "Allows ICMP connections from any source to any instance on the network."
            allow {
                protocol = "icmp"
            }
        }
      
        ## SSH
        resource "google_compute_firewall" "dev-cicd-vpc-allow-ssh" {
            network     = google_compute_network.dev-cicd-vpc.name
            project     = var.gcp_project_id
            direction   = "INGRESS"
            priority    = 65534
            source_ranges = ["0.0.0.0/0"]
            name        = "dev-cicd-vpc-allow-ssh"
            description = "Allows TCP connections from any source to any instance on the network using port 22."
            allow {
                protocol = "tcp" 
                ports    = ["22"]
            }
        }
      
๐Ÿ’ก
To install the various DevOps tools, I used Bash scripts and leveraged GCP's startup script feature when creating VMs.

Project Repo

  1. Kubernetes Cluster Deployment: Establishing a robust Kubernetes (K8s) cluster as the foundation for the project's deployment needs. This cluster is thoroughly scanned and secured to ensure a resilient platform.

    • Create 3 VM's on GCP for Kubernetes (1 Master and 2 Slave Nodes) and configure Kubernetes on them.

        ## Master Node VM
        resource "google_compute_instance" "cluster-instances-node-master" {
            boot_disk {
                auto_delete = true
                device_name = "k8-cluster-nodes"
      
                initialize_params {
                image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
                size  = 25
                type  = "pd-balanced"
                }
      
                mode = "READ_WRITE"
            }
      
            can_ip_forward      = false
            deletion_protection = false
            enable_display      = false
      
            labels = {
                goog-ec-src = "vm_add-tf"
                node        = "master"
            }
      
            machine_type = "e2-medium"
      
            metadata = {
                startup-script = file("./scripts/masterVM.sh")
            }
      
            name = "cluster-instances-node-master"
      
            network_interface {
                access_config {
                network_tier = "STANDARD"
                }
      
                queue_count = 0
                stack_type  = "IPV4_ONLY"
                network     = google_compute_network.dev-cicd-vpc.name
            }
      
            scheduling {
                automatic_restart   = true
                on_host_maintenance = "MIGRATE"
                preemptible         = false
                provisioning_model  = "STANDARD"
            }
      
            service_account {
                email  = var.gcp_service_account_email
                scopes = ["https://www.googleapis.com/auth/cloud-platform"]
            }
      
            shielded_instance_config {
                enable_integrity_monitoring = true
                enable_secure_boot          = false
                enable_vtpm                 = true
            }
      
            tags = ["http-server", "https-server", "lb-health-check"]
            zone = "asia-south1-c"
        }
      
        ## Slave Node VM (2 nodes)
        resource "google_compute_instance" "cluster-instances-node-slave" {
            count = 2
      
            boot_disk {
                auto_delete = true
                device_name = "k8-cluster-nodes"
      
                initialize_params {
                image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
                size  = 25
                type  = "pd-balanced"
                }
      
                mode = "READ_WRITE"
            }
      
            can_ip_forward      = false
            deletion_protection = false
            enable_display      = false
      
            labels = {
                goog-ec-src = "vm_add-tf"
                node        = "slave0${count.index + 1}"
            }
      
            machine_type = "e2-medium"
      
            metadata = {
                startup-script = file("./scripts/slaveVM.sh")
            }
      
            name = "cluster-instances-node-slave0${count.index + 1}"
      
            network_interface {
                access_config {
                network_tier = "STANDARD"
                }
      
                queue_count = 0
                stack_type  = "IPV4_ONLY"
                network     = google_compute_network.dev-cicd-vpc.name
            }
      
            scheduling {
                automatic_restart   = true
                on_host_maintenance = "MIGRATE"
                preemptible         = false
                provisioning_model  = "STANDARD"
            }
      
            service_account {
                email  = var.gcp_service_account_email
                scopes = ["https://www.googleapis.com/auth/cloud-platform"]
            }
      
            shielded_instance_config {
                enable_integrity_monitoring = true
                enable_secure_boot          = false
                enable_vtpm                 = true
            }
      
            tags = ["http-server", "https-server", "lb-health-check"]
            zone = "asia-south1-c"
        }
      
  2. Essential Server Setup: Configuring and deploying essential servers to support the CI/CD processes. This includes:

    • Jenkins Server: For continuous integration and deployment automation.

        # Jenkins VM
        #-----------------#
      
        resource "google_compute_instance" "jenkins" {
            boot_disk {
                auto_delete = true
                device_name = "jenkins-vm"
      
                initialize_params {
                image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
                size  = 30
                type  = "pd-balanced"
                }
      
                mode = "READ_WRITE"
            }
      
            can_ip_forward      = false
            deletion_protection = false
            enable_display      = false
      
            labels = {
                goog-ec-src = "vm_add-tf"
                jenkins     = ""
            }
      
            machine_type = "e2-standard-2"
      
            metadata = {
                startup-script = file("./scripts/jenkins.sh")
            }
      
            name = "jenkins"
      
            network_interface {
                access_config {
                network_tier = "STANDARD"
                }
      
                queue_count = 0
                stack_type  = "IPV4_ONLY"
                network     = google_compute_network.dev-cicd-vpc.name
            }
      
            scheduling {
                automatic_restart   = true
                on_host_maintenance = "MIGRATE"
                preemptible         = false
                provisioning_model  = "STANDARD"
            }
      
            service_account {
                email  = var.gcp_service_account_email
                scopes = ["https://www.googleapis.com/auth/cloud-platform"]
            }
      
            shielded_instance_config {
                enable_integrity_monitoring = true
                enable_secure_boot          = false
                enable_vtpm                 = true
            }
      
            tags = ["http-server", "https-server", "lb-health-check"]
            zone = "asia-south1-c"
        }
      
    • SonarQube Server: For continuous code quality and security analysis.

        # Sonarqube VM
        #---------------#
        resource "google_compute_instance" "sonarqube" {
            boot_disk {
                auto_delete = true
                device_name = "k8-cluster-nodes"
      
                initialize_params {
                image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
                size  = 20
                type  = "pd-balanced"
                }
      
                mode = "READ_WRITE"
            }
      
            can_ip_forward      = false
            deletion_protection = false
            enable_display      = false
      
            labels = {
                goog-ec-src = "vm_add-tf"
                sonarqube   = ""
            }
      
            machine_type = "e2-medium"
      
            metadata = {
                startup-script = file("./scripts/sonarqube.sh")
            }
      
            name = "sonarqube"
      
            network_interface {
                access_config {
                network_tier = "STANDARD"
                }
      
                queue_count = 0
                stack_type  = "IPV4_ONLY"
                network     = google_compute_network.dev-cicd-vpc.name
            }
      
            scheduling {
                automatic_restart   = true
                on_host_maintenance = "MIGRATE"
                preemptible         = false
                provisioning_model  = "STANDARD"
            }
      
            service_account {
                email  = var.gcp_service_account_email
                scopes = ["https://www.googleapis.com/auth/cloud-platform"]
            }
      
            shielded_instance_config {
                enable_integrity_monitoring = true
                enable_secure_boot          = false
                enable_vtpm                 = true
            }
      
            tags = ["http-server", "https-server", "lb-health-check"]
            zone = "asia-south1-c"
        }
      
    • Monitoring Servers: To support system and application-level monitoring from the outset.

        # Monitoring VM
        #-----------------#
      
        resource "google_compute_instance" "monitor" {
            boot_disk {
                auto_delete = true
                device_name = "monitoring"
      
                initialize_params {
                image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
                size  = 20
                type  = "pd-balanced"
                }
      
                mode = "READ_WRITE"
            }
      
            can_ip_forward      = false
            deletion_protection = false
            enable_display      = false
      
            labels = {
                goog-ec-src = "vm_add-tf"
                monitor     = ""
            }
      
            machine_type = "e2-standard-2"
      
            metadata = {
                startup-script = file("./scripts/monitoring.sh")
            }
      
            name = "monitor"
      
            network_interface {
                access_config {
                network_tier = "PREMIUM"
                }
      
                queue_count = 0
                stack_type  = "IPV4_ONLY"
                network     = google_compute_network.dev-cicd-vpc.name
            }
      
            scheduling {
                automatic_restart   = true
                on_host_maintenance = "MIGRATE"
                preemptible         = false
                provisioning_model  = "STANDARD"
            }
      
            service_account {
                email  = var.gcp_service_account_email
                scopes = ["https://www.googleapis.com/auth/cloud-platform"]
            }
      
            shielded_instance_config {
                enable_integrity_monitoring = true
                enable_secure_boot          = false
                enable_vtpm                 = true
            }
      
            tags = ["http-server", "https-server", "lb-health-check"]
            zone = "asia-south1-c"
        }
      

Phase 2: Version Control and Notification System

In this phase, the focus is on version control and establishing a notification system to keep stakeholders informed:

  1. Version Control with GitHub:

    • Private Repository Setup: Source code is pushed to a private GitHub repository to ensure security and privacy.

    • Branching Strategy: Implementing a branching strategy (e.g., feature branches, development, main) to manage code changes efficiently.

  2. Mail Notification System:

    • Configuration: Setting up an email notification system to alert stakeholders about the project's progress and any critical issues.

    • Integration: Integrating the notification system with the CI/CD pipeline to provide real-time updates on build statuses, deployments, and alerts.

Phase 3: Building the CI/CD Pipeline

The CI/CD pipeline is the heart of the project, automating the deployment process and ensuring robust security measures:

  1. Pipeline Design and Implementation:

    • Jenkins Pipeline Configuration: Defining a declarative Jenkins pipeline (Jenkinsfile) that outlines the steps for building, testing, and deploying the application.
  2. Security Measures:

    • Code Quality Analysis with SonarQube: Integrating SonarQube to perform static code analysis and ensure code quality.

    • Vulnerability Scanning with Aqua Trivy: Scanning code dependencies and Docker images for vulnerabilities to enhance security.

  3. Automated Deployment:

    • Artifact Creation and Storage: Building the application artifacts (e.g., JAR, WAR) and securely storing them in Nexus Repository.

    • Container Image Build and Scan: Creating Docker container images and scanning them for vulnerabilities before deployment.

    • Kubernetes Deployment: Automating the deployment of container images to the Kubernetes cluster, ensuring a smooth transition from development to production.

Phase 4: Monitoring at Scale

In the final phase, system-level and application-level monitoring are implemented to maintain the health of the application and provide real-time insights:

  1. System Monitoring with Prometheus and Grafana:

    • Metric Collection: Prometheus collects metrics from various services and endpoints, providing detailed insights into system performance.

    • Visualization and Alerting: Grafana visualizes these metrics and sets up alerting mechanisms to notify of any performance issues or anomalies.

  2. Application Monitoring:

    • Endpoint Monitoring with Blackbox Exporter: Ensuring that web endpoints are monitored for availability and responsiveness.

    • Traffic and Performance Monitoring: Monitoring website traffic and application performance to ensure optimal user experience.

  3. Continuous Improvement:

    • Feedback Loop: Using monitoring data to identify areas for improvement and optimize the CI/CD pipeline and deployment processes continually.

    • Proactive Maintenance: Implementing proactive measures based on monitoring insights to prevent issues before they impact users.

Security Considerations

  • Pre-deployment Kubernetes Security: The Kubernetes cluster is hardened to minimize potential attack surfaces.

  • Pipeline as Code: Pipeline configuration (e.g., Jenkinsfile) is treated as code, enabling versioning and peer review.

  • Vulnerability Scanning: Scans at the code and image stages proactively identify issues.

  • Monitoring: Anomalies or suspicious behavior are detected early.

Benefits of a Secure CI/CD Pipeline

  • Enhanced Security Posture: Integration of security measures throughout the process mitigates risks.

  • Increased Confidence: Rigorous testing and security checks lead to more reliable releases.

  • Improved Agility: Automation streamlines development and reduces time to market.

  • Reduced Operational Overhead: Fewer manual processes minimize potential errors.

Conclusion

This project exemplifies a comprehensive CI/CD pipeline that can serve as a blueprint for organizations looking to implement or refine their own CI/CD strategies. It demonstrates the importance of each phase and the inter-connectedness of various tools and practices that contribute to a successful software delivery lifecycle.

Did you find this article valuable?

Support Chetan Thapliyal by becoming a sponsor. Any amount is appreciated!

ย