diff --git a/docs/infrastructure/02-getting-started/basic-knowledge/Terraform.md b/docs/infrastructure/02-getting-started/basic-knowledge/Terraform.md deleted file mode 100644 index e69de29..0000000 diff --git a/docs/infrastructure/02-getting-started/basic-knowledge/_category_.yml b/docs/infrastructure/02-getting-started/basic-knowledge/_category_.yml deleted file mode 100644 index 78d023a..0000000 --- a/docs/infrastructure/02-getting-started/basic-knowledge/_category_.yml +++ /dev/null @@ -1,2 +0,0 @@ -label: Basic Knowledge -position: 2 diff --git a/docs/infrastructure/02-getting-started/basic-knowledge/index.md b/docs/infrastructure/02-getting-started/basic-knowledge/index.md deleted file mode 100644 index 53b663b..0000000 --- a/docs/infrastructure/02-getting-started/basic-knowledge/index.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -title: Basic Knowledge ---- -import DocCardList from '@theme/DocCardList'; - -## Overview - -This section explains how we organized our infrastructure. - -Everything regarding the deployment of apps and services is divided into two repositories: - -- [polinetworkorg/terraform](https://github.com/polinetworkorg/terraform) - Contains the Terraform code to deploy the infrastructure. This defines the resources that are created on Azure, from specific things like KeyVaults, to the Kubernetes cluster itself. -- [polinetworkorg/polinetwork-cd](https://github.com/polinetworkorg/polinetwork-cd) - Our GitOps repo, contains the manifests that define deployments specific for each app or service. This repository is connected to the ArgoCD instance running on the Kubernetes cluster. - -```mermaid -flowchart TD - subgraph Github - tf([**terraform**]) - cd([**polinetwork-cd**]) - end - subgraph Azure - vm[VMs and other resources] - subgraph k8s[K8S cluster] - argocd[ArgoCD] - app@{ shape: processes, label: "Apps / Services" } - end - end - - tf -.-> vm - tf -.-> k8s - vm <--> k8s - cd <-.-> argocd --> app -``` - - - diff --git a/docs/infrastructure/02-getting-started/cloud-infrastructure/01-terraform.md b/docs/infrastructure/02-getting-started/cloud-infrastructure/01-terraform.md new file mode 100644 index 0000000..1fcf7d3 --- /dev/null +++ b/docs/infrastructure/02-getting-started/cloud-infrastructure/01-terraform.md @@ -0,0 +1,255 @@ +--- +title: "Terraform" +description: "Detailed overview of our Azure-based infrastructure managed via the polinetworkorg/terraform repository." +--- + +## What is Terraform? + +Terraform is an open-source Infrastructure as Code (IaC) tool developed by +HashiCorp. It allows you to define, provision, and manage the infrastructure +using a declarative configuration language. This means you describe the desired +state of the infrastructure, and Terraform takes care of creating and updating +the resources to match that state. + +:::important +This documentation assumes you already have the Terraform CLI installed on your +local machine. If you need help setting up Terraform, please refer to the +[official documentation](https://learn.hashicorp.com/tutorials/terraform/install-cli). + +You'll also need to be logged in with the Azure CLI and have the necessary +permissions to manage the resources in our Azure subscription (more on this +[here](../setup#azure-cli--kubectl)). + +Once you have logged in to the Azure CLI, you need to set the necessary +environment variables by running this command at the root of the repository: + +```sh +source ./access_key.sh +``` + +More about this in the [repository README](https://github.com/PoliNetworkOrg/terraform/?tab=readme-ov-file#notes). +::: + +### Why We Use Terraform + +Our organization leverages Terraform to manage our Azure cloud infrastructure for several important reasons: + +- **Consistency:** Defining our infrastructure as code ensures that every environment is configured the same way, reducing the chance of human error. +- **Automation:** Terraform automates the provisioning and modification of resources, streamlining our deployment processes. +- **Version Control:** Storing Terraform configurations in version control systems allows us to track changes, collaborate effectively, and revert to previous versions if needed. +- **Modularity:** Terraform encourages breaking down the infrastructure into reusable modules, which simplifies management and enhances maintainability. + +### How Terraform Works + +:::warning +To run terraform commands you need to have the right permissions and credentials. +If you are not sure about your permissions, please contact the IT department (but +if you have to ask you probably shouldn't run `apply` yourself anyway). + +If you haven't yet set up the Azure CLI please refer to the [setup guide](../setup#azure-cli--kubectl). +::: + +Terraform operates through a simple yet powerful workflow: + +1. **Write:** You define the infrastructure in configuration files (typically with a `.tf` extension). +2. **Plan:** Running `terraform plan` generates an execution plan, showing you the changes Terraform will make to achieve the desired state. +3. **Apply:** Executing `terraform apply` implements the changes, creating or updating the infrastructure accordingly. + +:::danger +Running Terraform commands can modify the cloud infrastructure. + +**THIS CAN INCUR COSTS OR CAUSE DOWNTIME AND PERMANENT DATA LOSS IF NOT DONE PROPERLY.** + +Always review the execution plan (`terraform plan`) before applying changes. +::: + +### Terraform Code vs. Terraform State vs. Azure State + +- **Terraform Code:** + This consists of the configuration files that describe your desired + infrastructure. These files are the blueprint for what you want the infrastructure to look like. + +- **Terraform State:** + The state file (which we named `state.tfstate`) records the current state + of the infrastructure as managed by Terraform. This file is critical because + it maps your configuration to the real-world resources, tracks dependencies, + and allows Terraform to detect what needs to change during updates. Our state + file is stored remotely (in an Azure Storage Account) and is locked using + Azure's integrated leasing mechanism to prevent concurrent modifications. + +- **Azure State:** + This is the actual state of resources deployed in the Azure environment. + While Terraform keeps track of these resources via its state file, the real + configuration and runtime details exist within Azure. The Terraform state acts + as an intermediary, ensuring that the desired state (from the Terraform code) + and the actual state (in Azure) are in sync. If resources are modified directly + in Azure without updating Terraform, discrepancies may occur, highlighting the + importance of using Terraform as the single source of truth. + +This clear separation between code, Terraform state, and the actual cloud state +is key to maintaining a reliable and predictable infrastructure. + +## Terraform Implementation Overview + +Welcome to the documentation for the **polinetworkorg/terraform** repository. +This page provides an in-depth look at our Terraform implementation on +**Microsoft Azure**, detailing the architecture of our infrastructure and the +key modules that make it up. While this overview highlights our main components, +it’s important to note that there are alternative approaches to similar problems, +and our setup is tailored to our current needs. + +### Repository Overview + +The **polinetworkorg/terraform** repository is our centralized solution for +provisioning and managing cloud infrastructure on Azure. It follows a modular +design that breaks down the overall architecture into focused components. To give +you a rough idea of what is in there, here are some of the primary modules: + +- **aks:** Defines all resources related to our Azure Kubernetes Service (AKS) instance. +- **mariadb:** Manages the main MariaDB database. +- **monitoring:** Sets up our monitoring stack using Grafana and Prometheus. +- **argocd:** Configures our ArgoCD installation for GitOps-driven deployments. +- **storage:** Handles the Azure Storage Account where the Terraform state is stored. + +There are also additional modules in the repository that handle specific +applications or services, but these are not detailed in this general overview. + +:::info +While the choices made in this implementation reflect our specific requirements +and constraints, it's worth noting that other approaches exist for managing +cloud infrastructure with Terraform. + +Our modular design, state management strategy, +and access controls are tailored to our current operational needs and funding +situationβ€”but they are not set in stone. Please feel free to reach out to the +IT department if you have any questions or suggestions. +::: + +### Infrastructure Structure + +Our repository is organized to clearly separate concerns and promote reusability. +While the exact folder structure may vary, a typical layout looks like this: + +```bash +πŸ“ polinetworkorg/terraform/ + β”œβ”€ πŸ“ modules/ # All modules are kept in a dedicated directory + β”‚ β”œβ”€ πŸ“ aks/ # Module for Azure Kubernetes Service resources + β”‚ β”œβ”€ πŸ“ mariadb/ # Module for the main MariaDB database + β”‚ β”œβ”€ πŸ“ monitoring/ # Module for monitoring (Grafana + Prometheus) + β”‚ β”œβ”€ πŸ“ argocd/ # Module for ArgoCD installation + β”‚ β”œβ”€ πŸ“ storage/ # Module for Azure Storage Account where the state.tfstate is kept + β”‚ └─ [other modules] # Additional modules for specific services + β”œβ”€β”€ main.tf # Main Terraform configuration file + β”œβ”€β”€ variables.tf # Variables definition file + β”œβ”€β”€ outputs.tf # Outputs definition file + β”œβ”€β”€ README.md # General repository documentation + └── [other files] +``` + +### State Management + +We use an **Azure Storage Account** for managing the Terraform state. The state +file (`state.tfstate`) is stored within a dedicated container named `terraform-state`, +as defined in our **storage** module. The integrated leasing mechanism provided +by the Azure Storage Account is used for state locking, ensuring that concurrent +Terraform operations do not conflict with each other. + +### Access and Authorization + +Given the critical nature of our infrastructure, modifications (e.g., running +`terraform apply`) are restricted to personnel with sufficient clearance. As an +association, any infrastructure changes require approval from either the Head of +the IT Department or the Directive Council. All authorization is managed through +**Microsoft's Entra ID system**, ensuring that only properly authenticated and +authorized users can perform these operations. + +### Funding and Nonprofit Considerations + +As a nonprofit organization, we benefit from a **Microsoft Azure Grant for Nonprofits**, +which provides us with a budget of **€2000 per year**. This support helps offset +our cloud costs and underpins our commitment to maintaining a robust and scalable +infrastructure. + +:::info +This is one of the core reasons we are structured as a nonprofit organization, and why +we use Azure as our cloud provider. + +You can find more about our benefits [here](https://www.microsoft.com/en-us/nonprofits/azure). + +If you have access to the adminorg account, you can check the current status of +our Azure grant [here](https://www.microsoftazuresponsorships.com/). +::: + +## Making Changes to the Infrastructure + +This section provides a basic guide on how to update our infrastructure using +Terraform and introduces some of the automated workflows that help us maintain +consistency and control. + +### Basic Workflow for Infrastructure Changes + +1. **Edit the Terraform Code:** + Modify the relevant configuration files to define your desired changes in the + infrastructure. + +2. **Review Your Changes Locally:** + Run `terraform plan` to generate an execution plan. This command will show + you what changes will be made to align the actual infrastructure with your + code. + +3. **Cost Estimation with Infracost:** + When you open a pull request, our **Infracost workflow** is triggered + automatically. This workflow calculates the estimated cost impact of your + changes and posts a report in the pull request comments, so you can review + any potential cost implications before merging. + :::tip + If you want to run Infracost locally, you can install it by following the + [official documentation](https://www.infracost.io/docs/). + ::: + +4. **Apply the Changes:** + After your changes have been reviewed and approved, merge the pull request + and run `terraform apply` to implement the changes. This command updates the + Azure resources to match the configuration defined in your code. + +5. **Drift Detection:** + A nightly **Drift workflow** runs to detect any discrepancies between the + current Terraform state and the actual state in Azure. If this workflow + detects a drift from what the `terraform plan` output would predict, it + automatically opens an issue for further investigation. + +### Basic Terraform CLI Commands + +- **`terraform init`** + Initializes the working directory containing the Terraform configuration files. + :::tip + This has to be run as soon as you clone the repository or when you pull changes + that include new modules or providers. + ::: + +- **`terraform plan`** + Generates an execution plan showing what changes will be applied to the infrastructure. + +- **`terraform apply`** + Applies the changes defined in your Terraform configuration to the cloud environment. + +- **`terraform destroy`** + Destroys the resources managed by Terraform, useful for tearing down test environments or decommissioning infrastructure. + + :::danger + If you ever happen to be as much as tempted of using `terraform destroy`, you + better be **SURE AS SHIT** about what you are doing. Make sure to review the + implications thoroughly before proceeding. + > *One does not simply destroy PoliNetwork with one command.* + ::: + +By following these steps and leveraging our automated workflows, you ensure that +infrastructure changes are managed in a consistent, cost-aware, and controlled +manner. + +## References + +- [Terraform Documentation](https://www.terraform.io/docs/index.html) +- [Microsoft Azure Documentation](https://docs.microsoft.com/en-us/azure/) +- [Infracost Documentation](https://www.infracost.io/docs/) +- [Terraform Repository](https://github.com/polinetworkorg/terraform) diff --git a/docs/infrastructure/02-getting-started/cloud-infrastructure/_category_.yml b/docs/infrastructure/02-getting-started/cloud-infrastructure/_category_.yml new file mode 100644 index 0000000..09f8381 --- /dev/null +++ b/docs/infrastructure/02-getting-started/cloud-infrastructure/_category_.yml @@ -0,0 +1,2 @@ +label: Cloud Infrastructure +position: 2 diff --git a/docs/infrastructure/02-getting-started/cloud-infrastructure/index.md b/docs/infrastructure/02-getting-started/cloud-infrastructure/index.md new file mode 100644 index 0000000..2b18ab0 --- /dev/null +++ b/docs/infrastructure/02-getting-started/cloud-infrastructure/index.md @@ -0,0 +1,81 @@ +--- +title: Cloud Infrastructure +--- +import DocCardList from '@theme/DocCardList'; + +## Overview + +This section explains how we organized our infrastructure. + +Everything regarding the deployment of apps and services is divided into two repositories: + +- [polinetworkorg/terraform](https://github.com/polinetworkorg/terraform) - Contains the Terraform code to deploy the infrastructure. This defines the resources that are created on Azure, from specific things like KeyVaults, Disks, and VMs, to the Kubernetes cluster itself. +- [polinetworkorg/polinetwork-cd](https://github.com/polinetworkorg/polinetwork-cd) - Our GitOps repo, contains the manifests that define deployments specific for each app or service. This repository is connected to the ArgoCD instance running on the Kubernetes cluster. + +```mermaid +flowchart TD + subgraph Github + tf([**terraform**]) + cd([**polinetwork-cd**]) + end + subgraph Azure + vm[VMs and other resources] + subgraph k8s[K8S cluster] + argocd[ArgoCD] + app@{ shape: processes, label: "Apps / Services" } + end + end + + tf -.-> vm + tf -.-> k8s + vm <--> k8s + cd <-.-> argocd --> app +``` + +### **Terraform: Definition and Purpose** + +The **Terraform** repository is responsible for managing and provisioning cloud infrastructure. In simple terms, it defines and creates the resources needed for the infrastructure, such as: + +- **Networks and subnets** (Virtual Networks, Subnets, Security Groups) +- **Virtual machines** (VMs, Containers, Kubernetes Clusters) +- **Databases** (MariaDB, PostgreSQL) +- **Storage** (Blob Storage, Virtual Disks, File Shares) +- **Security services** (Identity and Access Management, Firewalls) + +The code in **Terraform** describes these resources declaratively: it defines the desired state, and Terraform applies the necessary changes to reach that state. This repository is primarily used to **create and modify the cloud infrastructure** where applications will run. + +### **PoliNetwork-CD: Definition and Purpose** + +The **PoliNetwork-CD** (Continuous Deployment) repository, on the other hand, is dedicated to managing and updating applications once the infrastructure has been created. Specifically, it uses **GitOps** tools (in our case, **ArgoCD**, but **Flux** is also an opinionated option) to: + +- **Define application deployments** in Kubernetes (or Kustomize, Helm, etc.) +- **Automatically update applications** when code changes +- **Manage application configurations** (e.g., configuration files, environment variables) +- **Monitor the state of applications** and ensure they remain in sync with the repository + +While Terraform is responsible for infrastructure, **PoliNetwork-CD** focuses on **managing the lifecycle of applications** running on that infrastructure, ensuring that code changes are applied automatically and securely. + +### **Key Difference: Infrastructure vs. Deployment** + +| Repository | Main Purpose | Example of Managed Resources | +|------------|--------------|------------------------------| +| **Terraform** | Creates and manages cloud infrastructure | Networks, VMs, databases, storage, security | +| **PoliNetwork-CD** | Automates application deployment and updates | Deployments, configurations, GitOps updates | + +In summary, **Terraform builds the foundation on which applications run**, while **PoliNetwork-CD manages the deployment and continuous updating of applications**. Both repositories work together to ensure a scalable and efficient cloud infrastructure. + + + + +## References + +### **Links to Repositories** + +- [Terraform Repository](https://github.com/polinetworkorg/terraform) +- [PoliNetwork-CD Repository](https://github.com/polinetworkorg/polinetwork-cd) + +### **Additional Resources** + +- [Kubernetes Documentation](https://kubernetes.io/docs/) +- [Terraform Documentation](https://www.terraform.io/docs/) +- [ArgoCD Documentation](https://argo-cd.readthedocs.io/en/stable/)