In today's software development landscape, the swift delivery of new features and updates is paramount; however, this must be balanced against the increasing importance of robust security practices. CI/CD (Continuous Integration/Continuous Delivery or Deployment) pipelines provide a framework for automating the building, testing, and deployment of applications, bolstering both speed and reliability. This project demonstrates a CI/CD pipeline where security is integrated as a first-class citizen throughout every stage.
Project Overview
The core objective of this project is to establish a CI/CD pipeline that prioritizes the following principles:
Security by Design: Security considerations are embedded in all phases of the development and deployment workflow.
Automation: Leveraging automation to maximize efficiency, reduce potential human error, and enforce security best practices.
Continuous Monitoring: Implementing systems and application-level monitoring for proactive issue detection and rapid response.
Infrastructure as Code with Terraform: Utilizing Terraform, a popular Infrastructure as Code (IaC) tool, to predictably create, change, and improve cloud infrastructure.
Google Cloud as the Cloud Platform: Leveraging Google Cloud's compute engine services and products to provision and manage the required infrastructure components.
Kubernetes: Container orchestration for application deployment and management.
Jenkins: CI/CD automation server.
SonarQube: Static code analysis to ensure code quality and identify potential security issues.
Aqua Trivy: Vulnerability scanning for code dependencies and container images.
Nexus Repository: Secure storage for build artifacts.
Docker: Containerization for application packaging.
Docker Hub: Docker image registry.
Kubeaudit: Tool to audit Kubernetes clusters for various different security concerns.
Grafana: For system and application-level monitoring and alerting.
Prometheus: For collecting and querying metrics from services and endpoints.
Gmail: For status notifications and alerts.
Workflow
A client requests a feature or change in the application and creates a Jira ticket. The Jira ticket is then assigned to a developer.
Development and Version Control:
Developers work on feature branches within a Git repository (e.g., GitHub) and successfully test the new feature in local environment.
Once changes are pushed along with source code to GitHub repository, it trigger the CI/CD pipeline.
Build and Unit Testing:
The project's build system (e.g., Maven) compiles the code, and find out if there are any syntax base errors in the code.
Once code compilation is successful, Unit tests are executed to validate code functionality.
Code Quality and Security Analysis:
SonarQube analyzes code for maintainability, potential bugs, and security vulnerabilities.
Aqua Trivy scans for vulnerabilities within project dependencies.
Artifact Creation and Storage:
A build artifact (e.g., JAR, WAR) is generated.
The artifact is pushed to Nexus Repository for secure storage and proper release.
Docker Image Creation:
Docker builds a container image incorporating the build artifact and tag it properly.
Aqua Trivy scans the image for vulnerabilities.
Docker image is pushed to dockerhub.
Kubernetes Deployment:
Mail Notification
Client and devops engineer will receive the mail notification if pipeline is successful or failed.
Notifications are sent to mail for deployment status, errors, and critical alerts.
Monitoring:
Monitoring tools (Prometheus and Grafana) track the health of the system and application.
For system-level monitoring of hardware, Jenkins is used. Node Exporter monitors Jenkins.
Step-by-Step Project Execution
Phase 1: Creating Infrastructure
The project kicks off with the creation of an isolated network environment a critical step for maintaining security and control. This phase includes:
Network Environment Creation: Setting up a dedicated and secure network environment to isolate and protect the project's infrastructure from external threats.
Create a Custom VPC on Google Cloud Platform with custom firewall rules allowing traffic to specific ports.
resource "google_compute_network" "dev-cicd-vpc" {
auto_create_subnetworks = true
description = "VPC for secure CICD pipeline."
mtu = 1460
name = "dev-cicd-vpc"
project = var.gcp_project_id
routing_mode = "REGIONAL"
}
resource "google_compute_firewall" "dev-cicd-vpc-allow-custom" {
name = "dev-cicd-vpc-allow-custom"
project = var.gcp_project_id
network = google_compute_network.dev-cicd-vpc.name
description = "Allows connection from any source to any instance on the network using custom protocols."
direction = "INGRESS"
priority = 65534
source_ranges = ["10.128.0.0/9", "0.0.0.0/0"]
allow {
protocol = "tcp"
ports = ["80", "443", "465", "6443", "3000-10000", "30000-32767"]
}
}
resource "google_compute_firewall" "dev-cicd-vpc-allow-icmp" {
network = google_compute_network.dev-cicd-vpc.name
project = var.gcp_project_id
direction = "INGRESS"
priority = 65534
source_ranges = ["0.0.0.0/0"]
name = "dev-cicd-vpc-allow-icmp"
description = "Allows ICMP connections from any source to any instance on the network."
allow {
protocol = "icmp"
}
}
resource "google_compute_firewall" "dev-cicd-vpc-allow-ssh" {
network = google_compute_network.dev-cicd-vpc.name
project = var.gcp_project_id
direction = "INGRESS"
priority = 65534
source_ranges = ["0.0.0.0/0"]
name = "dev-cicd-vpc-allow-ssh"
description = "Allows TCP connections from any source to any instance on the network using port 22."
allow {
protocol = "tcp"
ports = ["22"]
}
}
💡
To install the various DevOps tools, I used Bash scripts and leveraged GCP's startup script feature when creating VMs.
Project Repo
Kubernetes Cluster Deployment: Establishing a robust Kubernetes (K8s) cluster as the foundation for the project's deployment needs. This cluster is thoroughly scanned and secured to ensure a resilient platform.
Create 3 VM's on GCP for Kubernetes (1 Master and 2 Slave Nodes) and configure Kubernetes on them.
resource "google_compute_instance" "cluster-instances-node-master" {
boot_disk {
auto_delete = true
device_name = "k8-cluster-nodes"
initialize_params {
image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
size = 25
type = "pd-balanced"
}
mode = "READ_WRITE"
}
can_ip_forward = false
deletion_protection = false
enable_display = false
labels = {
goog-ec-src = "vm_add-tf"
node = "master"
}
machine_type = "e2-medium"
metadata = {
startup-script = file("./scripts/masterVM.sh")
}
name = "cluster-instances-node-master"
network_interface {
access_config {
network_tier = "STANDARD"
}
queue_count = 0
stack_type = "IPV4_ONLY"
network = google_compute_network.dev-cicd-vpc.name
}
scheduling {
automatic_restart = true
on_host_maintenance = "MIGRATE"
preemptible = false
provisioning_model = "STANDARD"
}
service_account {
email = var.gcp_service_account_email
scopes = ["https://www.googleapis.com/auth/cloud-platform"]
}
shielded_instance_config {
enable_integrity_monitoring = true
enable_secure_boot = false
enable_vtpm = true
}
tags = ["http-server", "https-server", "lb-health-check"]
zone = "asia-south1-c"
}
resource "google_compute_instance" "cluster-instances-node-slave" {
count = 2
boot_disk {
auto_delete = true
device_name = "k8-cluster-nodes"
initialize_params {
image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
size = 25
type = "pd-balanced"
}
mode = "READ_WRITE"
}
can_ip_forward = false
deletion_protection = false
enable_display = false
labels = {
goog-ec-src = "vm_add-tf"
node = "slave0${count.index + 1}"
}
machine_type = "e2-medium"
metadata = {
startup-script = file("./scripts/slaveVM.sh")
}
name = "cluster-instances-node-slave0${count.index + 1}"
network_interface {
access_config {
network_tier = "STANDARD"
}
queue_count = 0
stack_type = "IPV4_ONLY"
network = google_compute_network.dev-cicd-vpc.name
}
scheduling {
automatic_restart = true
on_host_maintenance = "MIGRATE"
preemptible = false
provisioning_model = "STANDARD"
}
service_account {
email = var.gcp_service_account_email
scopes = ["https://www.googleapis.com/auth/cloud-platform"]
}
shielded_instance_config {
enable_integrity_monitoring = true
enable_secure_boot = false
enable_vtpm = true
}
tags = ["http-server", "https-server", "lb-health-check"]
zone = "asia-south1-c"
}
Essential Server Setup: Configuring and deploying essential servers to support the CI/CD processes. This includes:
Jenkins Server: For continuous integration and deployment automation.
resource "google_compute_instance" "jenkins" {
boot_disk {
auto_delete = true
device_name = "jenkins-vm"
initialize_params {
image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
size = 30
type = "pd-balanced"
}
mode = "READ_WRITE"
}
can_ip_forward = false
deletion_protection = false
enable_display = false
labels = {
goog-ec-src = "vm_add-tf"
jenkins = ""
}
machine_type = "e2-standard-2"
metadata = {
startup-script = file("./scripts/jenkins.sh")
}
name = "jenkins"
network_interface {
access_config {
network_tier = "STANDARD"
}
queue_count = 0
stack_type = "IPV4_ONLY"
network = google_compute_network.dev-cicd-vpc.name
}
scheduling {
automatic_restart = true
on_host_maintenance = "MIGRATE"
preemptible = false
provisioning_model = "STANDARD"
}
service_account {
email = var.gcp_service_account_email
scopes = ["https://www.googleapis.com/auth/cloud-platform"]
}
shielded_instance_config {
enable_integrity_monitoring = true
enable_secure_boot = false
enable_vtpm = true
}
tags = ["http-server", "https-server", "lb-health-check"]
zone = "asia-south1-c"
}
SonarQube Server: For continuous code quality and security analysis.
resource "google_compute_instance" "sonarqube" {
boot_disk {
auto_delete = true
device_name = "k8-cluster-nodes"
initialize_params {
image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
size = 20
type = "pd-balanced"
}
mode = "READ_WRITE"
}
can_ip_forward = false
deletion_protection = false
enable_display = false
labels = {
goog-ec-src = "vm_add-tf"
sonarqube = ""
}
machine_type = "e2-medium"
metadata = {
startup-script = file("./scripts/sonarqube.sh")
}
name = "sonarqube"
network_interface {
access_config {
network_tier = "STANDARD"
}
queue_count = 0
stack_type = "IPV4_ONLY"
network = google_compute_network.dev-cicd-vpc.name
}
scheduling {
automatic_restart = true
on_host_maintenance = "MIGRATE"
preemptible = false
provisioning_model = "STANDARD"
}
service_account {
email = var.gcp_service_account_email
scopes = ["https://www.googleapis.com/auth/cloud-platform"]
}
shielded_instance_config {
enable_integrity_monitoring = true
enable_secure_boot = false
enable_vtpm = true
}
tags = ["http-server", "https-server", "lb-health-check"]
zone = "asia-south1-c"
}
Monitoring Servers: To support system and application-level monitoring from the outset.
resource "google_compute_instance" "monitor" {
boot_disk {
auto_delete = true
device_name = "monitoring"
initialize_params {
image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
size = 20
type = "pd-balanced"
}
mode = "READ_WRITE"
}
can_ip_forward = false
deletion_protection = false
enable_display = false
labels = {
goog-ec-src = "vm_add-tf"
monitor = ""
}
machine_type = "e2-standard-2"
metadata = {
startup-script = file("./scripts/monitoring.sh")
}
name = "monitor"
network_interface {
access_config {
network_tier = "PREMIUM"
}
queue_count = 0
stack_type = "IPV4_ONLY"
network = google_compute_network.dev-cicd-vpc.name
}
scheduling {
automatic_restart = true
on_host_maintenance = "MIGRATE"
preemptible = false
provisioning_model = "STANDARD"
}
service_account {
email = var.gcp_service_account_email
scopes = ["https://www.googleapis.com/auth/cloud-platform"]
}
shielded_instance_config {
enable_integrity_monitoring = true
enable_secure_boot = false
enable_vtpm = true
}
tags = ["http-server", "https-server", "lb-health-check"]
zone = "asia-south1-c"
}
Phase 2: Version Control and Notification System
In this phase, the focus is on version control and establishing a notification system to keep stakeholders informed:
Version Control with GitHub:
Private Repository Setup: Source code is pushed to a private GitHub repository to ensure security and privacy.
Branching Strategy: Implementing a branching strategy (e.g., feature branches, development, main) to manage code changes efficiently.
Mail Notification System:
Configuration: Setting up an email notification system to alert stakeholders about the project's progress and any critical issues.
Integration: Integrating the notification system with the CI/CD pipeline to provide real-time updates on build statuses, deployments, and alerts.
Phase 3: Building the CI/CD Pipeline
The CI/CD pipeline is the heart of the project, automating the deployment process and ensuring robust security measures:
Pipeline Design and Implementation:
- Jenkins Pipeline Configuration: Defining a declarative Jenkins pipeline (Jenkinsfile) that outlines the steps for building, testing, and deploying the application.
Security Measures:
Code Quality Analysis with SonarQube: Integrating SonarQube to perform static code analysis and ensure code quality.
Vulnerability Scanning with Aqua Trivy: Scanning code dependencies and Docker images for vulnerabilities to enhance security.
Automated Deployment:
Artifact Creation and Storage: Building the application artifacts (e.g., JAR, WAR) and securely storing them in Nexus Repository.
Container Image Build and Scan: Creating Docker container images and scanning them for vulnerabilities before deployment.
Kubernetes Deployment: Automating the deployment of container images to the Kubernetes cluster, ensuring a smooth transition from development to production.
Phase 4: Monitoring at Scale
In the final phase, system-level and application-level monitoring are implemented to maintain the health of the application and provide real-time insights:
System Monitoring with Prometheus and Grafana:
Metric Collection: Prometheus collects metrics from various services and endpoints, providing detailed insights into system performance.
Visualization and Alerting: Grafana visualizes these metrics and sets up alerting mechanisms to notify of any performance issues or anomalies.
Application Monitoring:
Endpoint Monitoring with Blackbox Exporter: Ensuring that web endpoints are monitored for availability and responsiveness.
Traffic and Performance Monitoring: Monitoring website traffic and application performance to ensure optimal user experience.
Continuous Improvement:
Feedback Loop: Using monitoring data to identify areas for improvement and optimize the CI/CD pipeline and deployment processes continually.
Proactive Maintenance: Implementing proactive measures based on monitoring insights to prevent issues before they impact users.
Security Considerations
Pre-deployment Kubernetes Security: The Kubernetes cluster is hardened to minimize potential attack surfaces.
Pipeline as Code: Pipeline configuration (e.g., Jenkinsfile) is treated as code, enabling versioning and peer review.
Vulnerability Scanning: Scans at the code and image stages proactively identify issues.
Monitoring: Anomalies or suspicious behavior are detected early.
Benefits of a Secure CI/CD Pipeline
Enhanced Security Posture: Integration of security measures throughout the process mitigates risks.
Increased Confidence: Rigorous testing and security checks lead to more reliable releases.
Improved Agility: Automation streamlines development and reduces time to market.
Reduced Operational Overhead: Fewer manual processes minimize potential errors.
Conclusion
This project exemplifies a comprehensive CI/CD pipeline that can serve as a blueprint for organizations looking to implement or refine their own CI/CD strategies. It demonstrates the importance of each phase and the inter-connectedness of various tools and practices that contribute to a successful software delivery lifecycle.
