Secure TERRAFORM Deployment on AWS with GITLAB-CI and VAULT - Part 3 - Devoteam A Cloud (2023)

How do you secure Terraform deployments on AWS with Gitlab-CI and Vault? In previous articles we have seenIssues with CI/CD deployed in the cloudand how to solve these problemsUse Vault to dynamically generate secrets and validate Gitlab-CI pipelinesIn this third and final article, we will discuss secret recovery from the application side.

In the previous section, we saw how to solve various problems encountered in the process stage. But we use Gitlab CI is not limited to infrastructure implementation but also includes the application itself.

Summarizing what we have done so far at the workflow level, we have the following diagram:

Secure TERRAFORM Deployment on AWS with GITLAB-CI and VAULT - Part 3 - Devoteam A Cloud (1)

As we can see, our application needs to retrieve the secrets stored in the Vault to connect to the database.

However, we want the interaction with the app and the vault to be as seamless as possible, and for that we need to reduce dependencies:

  • About certification: Which method should we choose to make our application as transparent and secure as possible?
  • About using secret keys: How to return dynamic secrets (short TTL) without affecting application code?

authentication application

When it comes to authenticating our apps using Vault, if we want it to be as seamless and secure as possible, it's important to base it on the environment where our app is deployed.

Our application is deployed on AWS, which is perfect because on the Vault side we haveAWS-style authentication methods.

There are two types of authentication:I amIEC2.

Since our application is deployed on an EC2 instance, we will use the EC2 type.

If we take a closer look at how this method works, we come across the following:

Secure TERRAFORM Deployment on AWS with GITLAB-CI and VAULT - Part 3 - Devoteam A Cloud (2)
  • 0 - So far Gitlab-CI has deployed our application to EC2 instances via Terraform and stored the database secrets in Vault.
  • 1 – Once our EC2 instance is deployed, its metadata (such as instance ID, subnet, VPC ID where our EC2 instance is deployed, etc.) is obtained via the EC2 metadata service. You can findOfficial AWS Documentation.
  • 2 - Our app is verified in Vault using AWS EC2 methodg PKCS7 signature.
  • 3 – Vault authenticates and the EC2 instance hosting our application meets the authentication requirements (binding parameters) (eg is it in the correct VPC and subnet? is it the correct instance ID? etc.)
  • 4 – If authentication is successful, Vault returns a token.

To implement this authentication method, you must:

  • Vault must be able to verify the identity and metadata of an EC2 instanceto the target AWS account.

Specify authentication requirements (binding parameters) in the vault for which we want to allow the application to authenticate.

Identity and metadata verification

In order for Vault to view the information on our EC2 instance, it needs permission from the target account to describe the example as follows:ec2: Describes the case.

To do this, and following the same logic as in our previous demo, we'll create an IAM role that Vault will assume.

The IAM role must contain the following rules:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeInstances" ], "Resource": "*" } ]}

Irelationship of trustAuthorize the source account (the account where the vault resides) to assume the IAM role. Regarding the configuration of the vault,We configure it through Terraform:

resource "vault_auth_backend" "aws" { description = "Auth backend to auth project in AWS env" type = "aws" path = "${var.project_name}-aws"}πόρος "vault_aws_auth_backend_sts_role" "role" { backend = vault_auth_backend. aws.path account_id = split(":", var.vault_aws_assume_role)[4] sts_role = var.vault_aws_assume_role}

As we can see, we also define the IAM role (STS) that Vault should assume.

At this point Vault can verify the information on our EC2 instance.

Set authentication constraints (binding parameters)

In terms of security, under what conditions do we allow our app to authenticate to the Vault to retrieve its secrets?

If we take a closer look at Vault's EC2-style AWS authentication approach, we can base it onvarious standards, For example:AMI ID, Account ID, Region, VPC ID, Subnet ID, and IAM Role ARN,ARN presence profiletheEC2 instance ID.

We can specify multiple criteria and multiple values ​​for each criterion. The value of each condition must be met for Vault to accept authentication.

In our case, our EC2 instances were created by Terraform. This is ideal because with Terraform we can retrieve all the snapshot properties and set them on the Treasury side asbinding parameters.which gives usNext Terraform clipCreate a Vault role for the AWS authentication backend:

资源 "vault_aws_auth_backend_role" "web" { backend = local.aws_backend role = var.project_name auth_type = "ec2" bound_ami_ids = [] bound_account_ids.ids. s = [] bound_subnet_ids = [aws_instance.web.subnet_id] bound_region = var.region token_ttl = var.project_token_ttl token_max_ttl = var.project_token_max_ttl",]project_token_max_ttl"]

In our case, we depend onAMI identifier,VPC number,subnet numberIAWS region.We could add an instance ID to strengthen the security of our authentication, but this pattern should be avoidedAuto Scaling Group.

At this point, our application can authenticate and retrieve its secret from the vault.

Secrets of using our application

On the Vault integration side of our application we will usecashier.

For those who want to know more details about Vault Agent integration, you can refer toThe articles below.

Let's take a closer look at our application workflow using Vault:

Secure TERRAFORM Deployment on AWS with GITLAB-CI and VAULT - Part 3 - Devoteam A Cloud (3)

As we can see, the Vault agent manages two phases:

  • AWSconfirmSpin using Vault and Vault Tokens
  • secretrecoveryand their tobacco

underFormation of the Ministry of FinanceWe have:

auto_auth { 方法 { mount_path = "auth/${vault_auth_path}" type = "aws" config = { type = "ec2" role = "web" } } sink { type = "file" config = { path = "/home/ ec2-user/.vault-token" } }}predložak { izvor = "/var/www/secrets.tpl" odredište = "/var/www/secrets.json"}

Terraform can configure this file to override certain values, e.g.the role of the fundname orinstallation path.The days,As for the secret standards, we want to get the secret in JSON format, which gives us the following format:

{{{ με μυστικό "web-db/creds/web" }} "username":"{{ .Data.username }}", "password":"{{ .Data.password }}", "db_host": "${db_host}", "db_name":"${db_name}" {{ kraj }}}

Vault is responsible for all things Vault related (Vault tokens, secrets, refreshes, etc.)Let our app retrieve its secrets from the related files:

if (file_exists("/var/www/secrets.json")) { $secrets_json = file_get_contents("/var/www/secrets.json", "r"); $user = json_decode($json_secrets)->{'my friend'}; $pass = json_decode($json_secrets)->{'password'}; $host = json_decode($json_secrets)->{'db_host'}; $dbname = json_decode($secrets_json)->{'db_name ' };}else{ echo "This is not trying.";exception?}

This gives us the expected result:

Secure TERRAFORM Deployment on AWS with GITLAB-CI and VAULT - Part 3 - Devoteam A Cloud (4)

Compared to what we do now, we have the following workflow:

Secure TERRAFORM Deployment on AWS with GITLAB-CI and VAULT - Part 3 - Devoteam A Cloud (5)
  • Terraform has a Vault service providerto simplify the interaction between the two tools.
  • Some pipeline branches or Gitlab-CI tasks can be certified through VaultJWT authentication method.
  • The Ministry of Finance allowsCloud provider credentials to createTo implement IaC through Terraform.for amazon, which can assume IAM roles across multiple AWS accounts, allowing our Terraform to implement IaC across multiple AWS accounts.
  • The Ministry of Finance allowsCollect more types of secretsProvides information to projects in an environmentally neutral manner, including secrets generated by the IAC.
  • Vault Agent simplifies vault integrationIn the application lifecycle that handles authentication and secrets
    • The Vault Token used by the Vault Agent is short-lived and changes frequently.
    • Application Secret (databaseIn our example) is short-lived and updated frequently via the Vault proxy.
  • We allow our app to authenticate to Vault based on the environment it's in.For our AWS application, we rely on AWS EC2 authentication andbinding parametersFor example, Subnet ID, VPC ID, AWS Region, etc. where our app is located.

As we saw in this article, Vault allows us to protect Terraform end-to-end through Gitlab-CI, including IaC or our application itself.

Additionally, the Vault proxy allows us to reduce the application's dependency on Vault.

When used correctly, this integration is transparent to operators, developers, and applications. The secret becomes transparent to allshort life cycle.why would you know the secretWhat if we could have transparent secrets as a service?

Author :Mehdi Laruelle, Devoteam Revolve Alumni


How to connect Terraform Cloud to GitLab? ›

Configuring Access
  1. Step 1: On Terraform Cloud, Begin Adding a New VCS Provider. Go to your organization's settings and then click Providers. ...
  2. Step 2: On GitLab, Create a New Application. ...
  3. Step 3: On Terraform Cloud, Set up Your Provider. ...
  4. Step 4: On Terraform Cloud, Set Up SSH Keypair (Optional)

How to implement CI CD pipeline using AWS and GitLab? ›

  1. Adding Runtime Environment variables on GitLab. ...
  2. Pushing the code to repo. ...
  3. Creating Pipeline from GitLab Runner. ...
  4. Create an EC2 instance to deploy the application and it can be public. ...
  5. Configure EC2 instance by installing necessary packages. ...
  6. Creating Pipeline and CodeDeploy. ...
  7. Create a Pipeline.

What does the terraform vault provider offer terraform users? ›

The Vault provider allows Terraform to read from, write to, and configure HashiCorp Vault. Interacting with Vault from Terraform causes any secrets that you read and write to be persisted in both Terraform's state file and in any generated plan files.

How do you deploy cloud resources using Terraform? ›

Deploy a basic Flask web server by using Terraform
  1. Create the directory.
  2. Create the Virtual Private Cloud network and subnet.
  3. Create the Compute Engine VM resource.
  4. Initialize Terraform.
  5. Validate the Terraform configuration.
  6. Apply the configuration.

How do I create a CI CD pipeline in AWS EC2 instance? ›

Building Your First CI/CD Pipeline on AWS
  1. Let's Start Deploying. First, you need an account with AWS. ...
  2. Example Application. ...
  3. AWS Requirements. ...
  4. Staying Organized. ...
  5. The AWS Management Console. ...
  6. Create an IAM User. ...
  7. Create an S3 Bucket. ...
  8. Create your EC2 Instance.

How do I create a CI CD pipeline using AWS? ›

CI/CD pipeline steps
  1. Pull the source code from source control.
  2. Lint any configuration files.
  3. Run unit tests against the AWS Lambda functions in codebase.
  4. Deploy the test pipeline.
  5. Run end-to-end tests against the test pipeline.
  6. Clean up test state machine and test infrastructure.
  7. Send approval to approvers.
Mar 2, 2020

How do you integrate Terraform and vault? ›

Run Terraform to configure Vault
  1. Optional: Start a Vault server in development mode with root as the root token if you don't have one running already. ...
  2. Set the client token in the VAULT_TOKEN environment variable. ...
  3. Initialize Terraform to pull Vault provider plugin. ...
  4. Execute the apply command to configure Vault.

How do you set up a Terraform vault? ›

To deploy a Vault Cluster:
  1. git clone this repo to your computer.
  2. Optional: build a Vault and Consul AMI. ...
  3. Install Terraform.
  4. Open , set the environment variables specified at the top of the file, and fill in any other variables that don't have a default. ...
  5. Run terraform init .
  6. Run terraform apply .

How do you use the key vault in Terraform? ›

Inside the keyvault folder, create the file to store variables used by the module: Then, create the to create the Azure Key Vault and policies, inside the keyvault folder: Finally, we create the file in the same folder used to return the values of the Terraform module.

How do I deploy infrastructure using Terraform in AWS? ›

To deploy infrastructure with Terraform:
  1. Scope - Identify the infrastructure for your project.
  2. Author - Write the configuration for your infrastructure.
  3. Initialize - Install the plugins Terraform needs to manage the infrastructure.
  4. Plan - Preview the changes Terraform will make to match your configuration.

How does Terraform deploy to AWS? ›

To deploy a Virtual Machine in AWS, we will need:
  1. AWS Credentials.
  2. Create an AWS Key Pair.
  3. Define AWS and Terraform Providers.
  4. Create a VPC, Subnet and other network components.
  5. Create Operating System Versions Variables.
  6. (Optional) Create a Bootstrapping script to install and/or configure applications.
Feb 23, 2022

How do I deploy an application in AWS EC2 using Terraform? ›

However, any text editor will work.
  1. Create the file. Open your text/code editor and create a new directory. ...
  2. Create the file. Once the file is created, it's time to set up the necessary variables. ...
  3. Create the EC2 environment. ...
  4. Clean up the environment.
Apr 20, 2022

How does Terraform know which AWS account to use? ›

The provider alias allows Terraform to differentiate the two AWS providers. To allow users in a different AWS account to assume a role, you must define an AssumeRole policy for that account. This configuration uses the aws_caller_identity data source to access the source account's ID.

How do I configure AWS credentials in Terraform cloud? ›

Configuring the integration requires the following steps:
  1. Configure AWS: Set up a trust configuration between AWS and Terraform Cloud. ...
  2. Configure Terraform Cloud: Add environment variables to the Terraform Cloud workspaces where you want to use Dynamic Credentials.

Does GitLab work with AWS? ›

GitLab is a DevOps platform with bring-your-own-infrastructure flexibility. From the on-premise to cloud, run GitLab on AWS and deploy to your workloads and AWS infrastructure using a single solution for everyone on your pipeline.

How to deploy REST API in AWS EC2? ›

You must have created a REST API before deploying it for the first time.
Sign in to the API Gateway console at .
  1. In the APIs navigation pane, choose the API you want to deploy.
  2. In the Resources navigation pane, choose Actions.
  3. From the Actions drop-down menu, choose Deploy API.

Can terraform be used for CI CD pipeline? ›

Terraform Cloud can be fully operated via API, CLI, and UI, which allows organizations to easily integrate it into their existing CI/CD pipelines, IT service management interfaces, and version control system processes.

How to build a CI CD pipeline in AWS in 5 minutes and 58 seconds? ›

Let's get started.
  1. Prerequisites. This pipeline uses the SAM template to build and deploy a serverless application. ...
  2. Create the Repository. ...
  3. The Initial Commit. ...
  4. Create an S3 Bucket. ...
  5. Create a Pipeline Role. ...
  6. Build the Pipeline. ...
  7. Update the Build Permissions. ...
  8. Run the Pipeline.
Aug 23, 2019

How do I deploy an application to AWS? ›

  1. Prerequisites.
  2. Step 1: Set up a GitHub account.
  3. Step 2: Create a GitHub repository.
  4. Step 3: Upload a sample application to your GitHub repository.
  5. Step 4: Provision an instance.
  6. Step 5: Create an application and deployment group.
  7. Step 6: Deploy the application to the instance.
  8. Step 7: Monitor and verify the deployment.

How do I make my CI CD pipeline secure? ›

The first steps for securing your team's CI/CD pipeline include locking down configuration managers, systems that host repositories and the build servers. The pipeline should be monitored from end to end with watertight access control across the entire toolchain.

What is the difference between AWS CodeDeploy and CodePipeline? ›

The main difference between AWS CodeDeploy and AWS CodePipeline is the type of application they deploy. AWS CodeDeploy is used to deploy applications to Amazon EC2 instances, Amazon ECS containers, or any other instance type supported by AWS. AWS CodePipeline is used to deploy applications to Amazon S3 buckets.

How do I set up a CI CD pipeline for AWS Lambda? ›

How to Set Up a CI/CD Pipeline for an AWS Lambda Function via Terraform
  1. Basic Requirements.
  2. Step 1: Setting Up the GitHub Repository.
  3. Step 2: Writing the Lambda Function.
  4. Step 3: Defining the Infrastructure with Terraform.
  5. Step 4: Writing the Workflow.
  6. Step 5: Running the Workflow.
Aug 9, 2022

How do you integrate Terraform clouds? ›

The Terraform Cloud Integration program has the following five steps.
  1. Engage: Sign up for the Technology Partner Program.
  2. Develop & Test: Understand and build using the API integration for Run Tasks.
  3. Review: Review integration with HashiCorp Alliances team.
  4. Release: Provide documentation for your Integration.

How to integrate Terraform with Git? ›

On this page:
  1. Automate Terraform with GitHub Actions.
  2. Prerequisites.
  3. Set up Terraform Cloud.
  4. Set up a GitHub repository.
  5. Review Actions workflows.
  6. Create pull request.
  7. Review and merge pull request.
  8. Verify EC2 instance provisioned.

What is the best way to store secrets Terraform? ›

File encryption is an effective technique to store and manage access keys and secrets. Terraform users can use this technique to encrypt sensitive information stored in the config and state files. This technique relies on: Encrypting the secrets.

Is Terraform vault free? ›

How do Terraform Cloud paid features work? Terraform Cloud is free to use but additional capabilities will be made available for purchase. These additional capabilities are offered as paid features, available for purchase on a per Organization basis.

Where is Terraform lock? ›

Lock File Location

For that reason Terraform creates it and expects to find it in your current working directory when you run Terraform, which is also the directory containing the . tf files for the root module of your configuration. The lock file is always named . terraform.

How is key vault secure? ›

Azure Key Vault is a secure secrets store, providing management for secrets, keys, and certificates, all backed by Hardware Security Modules.

How do I connect Terraform cloud to GitHub? ›

Open Terraform Cloud in your browser and navigate to your organization settings. Then, select "VCS Providers". Click the "Add VCS Provider" button. Select "GitHub" then " (Custom)" from the dropdown.

How to connect GitLab to Google Cloud? ›

Connecting to a GitLab Enterprise Edition repository
  1. Open the Triggers page in the Google Cloud console. ...
  2. Click Connect Repository. ...
  3. Under Select source, select GitLab Enterprise Edition.
  4. Under Select Repository, select the GitLab Enterprise Edition repositories you wish to connect to Cloud Build.

How to deploy VPC AWS using Terraform? ›

Running Terraform to Create the AWS VPC
  1. Open a terminal and navigate to the ~\terraform-vpc-demo directory. cd ~\terraform-vpc-demo. ...
  2. Run the terraform init command in the same directory. ...
  3. Now, run the terraform plan command. ...
  4. Next, tell Terraform actually to provision the AWS VPC and resources using terraform apply .
Apr 20, 2021

How to install terraform on aws cloud shell? ›

Clone tfenv by running this command :
  1. git clone ~/.tfenv mkdir ~/bin. Make sysmlink for tfenv/bin/* scripts into a path ~/bin because it has already added to $PATH.
  2. ln -s ~/.tfenv/bin/* ~/bin/ Install Terraform with tfenv . ...
  3. tfenv install 1.2.5. ...
  4. tfenv use 1.2.5. ...
  5. terraform --version.
Aug 26, 2022

What is the branching strategy of terraform cloud? ›

Trunk based development is the most recommended git branching strategy and you can apply it to your terraform code as well. You can achieve this by separating your deployment code into multiple directories and having different triggers or pipelines for every folder.

What is the difference between terraform and terraform Cloud? ›

Terraform Cloud vs.

Another option Hashicorp offers for Terraform Cloud is its Enterprise solution. The main difference is that Terraform Enterprise is self-hosted and offered as a private installation rather than a SaaS solution. Terraform Enterprise offers most of the same features as the Terraform Business tier.

What is the best CI CD tool for terraform? ›

It's recommended to use CI/CD tools like Jenkins, TravisCI, or CircleCI to automate these steps and integrate with version control systems like git to version control your terraform code.

Can terraform manage cross Cloud dependencies? ›

Terraform lets you use the same workflow to manage multiple providers and handle cross-cloud dependencies.

Can we connect GitLab to AWS? ›

The first thing that will come into everyone's mind is to simply store ready-to-use credentials of a technical user as part of Gitlab CI/CD Variables. Those will be provided to each step of your CI/CD pipeline (. gitlab-ci. yml) and you will be able to perform and deploy operations using AWS CLI or AWS SAM .

What cloud provider does GitLab use? ›

Our core infrastructure is primarily hosted in Google Cloud Platform's (GCP) us-east1 region (see Regions and Zones).

How to create a Terraform pipeline in GitLab? ›

How to Implement GitLab CI/CD Pipeline with Terraform
  1. Set up GitLab project repository.
  2. Create the Terraform configuration files.
  3. Set up pipelines using .gitlab-ci.yml file.
  4. Set up AWS Credentials in Gitlab.
  5. Set up the remote backend for Terraform on Gitlab.
  6. Configure the backend in the provider block for local development.
Mar 29, 2023

What is the difference between Terraform and GitLab? ›

GitLab can be classified as a tool in the "Code Collaboration & Version Control" category, while Terraform is grouped under "Infrastructure Build Tools". Some of the features offered by GitLab are: Manage git repositories with fine grained access controls that keep your code secure.


Top Articles
Latest Posts
Article information

Author: Rob Wisoky

Last Updated: 06/12/2023

Views: 5241

Rating: 4.8 / 5 (48 voted)

Reviews: 87% of readers found this page helpful

Author information

Name: Rob Wisoky

Birthday: 1994-09-30

Address: 5789 Michel Vista, West Domenic, OR 80464-9452

Phone: +97313824072371

Job: Education Orchestrator

Hobby: Lockpicking, Crocheting, Baton twirling, Video gaming, Jogging, Whittling, Model building

Introduction: My name is Rob Wisoky, I am a smiling, helpful, encouraging, zealous, energetic, faithful, fantastic person who loves writing and wants to share my knowledge and understanding with you.