GitOps and GitLab: How to update image tags in YAML via CI/CD pipeline (2023)

This is the first post in a longer series documenting my discoveries while on the farmGitLaband CI/CD (Continuous Development/Continuous Integration) solution for running Kubernetes clusters and their workloads while adhering to a GitOps approach.

For the GitOps setup I choseArgonautsky Disc, which is responsible for syncing my configuration from Git to Kubernetes. The different Kubernetes cluster workloads are configured in the following waysruddergraph. The rudder maps for my workload are varied, but most of them have one thing in common, which is that they require image tags as input. This image tag specifies which version of a particular docker container of my application will be deployed.

If I want to deploy a newer version, I need to update this image tag. It's fine to do it by hand occasionally, but an automated approach often makes it more scalable, especially for larger teams. A common use case is to automatically deploy whenever a branch is merged into the master branch, or if one wants to quickly test a merge request by deploying it in a staging environment.

To automate this, we can create a custom GitLab CI/CD pipeline. The CI/CD workflow must perform the following steps:

• to take on new goalsLet me underlineas an input variable
• See the repo that specifies the Helm maps
• processingvalue.yamlUpdate the desired tag variable for
• Create a new commit on the master branch of the repository
• Push the changes to the remote source branch

Let's imagine a Helm chart with the following examplevalue.yamldocument:

My application:
image:
Tags: 8nf3u23e

To update the YAML file in place, we'll use a file calledyq.is the YAML editor it usesto go.With the help of the following command, we can replace the tag value with this line of code:

yq -i '.myapp.image.tag = "my_new_tag"' values.yaml

For GitLab pipeline tasks we will use yq (mikefarah/yq:4).

To update the mirror tag, we not only update the file, but edit and merge the changes into the appropriate Git repository for the Helm graph. This requires a few extra steps in the GitLab CI/CD pipeline.

When the pipeline starts, the activation branch is copied to the working directory, but no remote origin is specified. Before we can push changes, we need to set it up. Private repositories require additional authorization for which the GitLab access token can be used. There are two possibilities:

• Personal Access Token: Can be created via account level profile settings here:https://gitlab.com/-/profile/personal_access_tokens.
  The following two fields (permissions) are required:read the repositoryIwriting warehouse
• Project Access Token: More information can be found athere.The required permissions are the same.

After creation we can add this token and token name asCI/CD variablesin the Helm charts repository. For our template we will useREPO_TOKEN_NAMEIREPO_ACCESS_TOKENas a variable name.

We can then set the remote origin in the pipeline as follows:

git remote set url origin "https://$REPO_TOKEN_NAME:$REPO_ACCESS_TOKEN@/.git"

I wantwith your GitLab domain (it should be gitlab.com if you're using the official GitLab SaaS) andwith your repository path, e.g.my-project/my-application.

Depending on your setup, there are different scenarios when and where this pipeline will run:

• A: You have a warehouseMy applicationWhere are the application code and Helm maps for development.
• B: You have storage spaceMy applicationLocation and repository where the application code residesmy mapWhere is the Helm chart that sets it up.

Based on scenarios A and B described earlier, we configured the pipeline differently. To support both scenarios, we will configure a generic and reusable funnel template. The template will take three variables as input:

• file path: the path to the file in the repository where the tag should be updated, for examplecharts/values.yaml
• Let me underline: A new image tag that should replace an existing image tag, for example1.0.3
• route sign: The path to the field in the YAML file to be updated, for example.myapp.image.tagIvalue.yamlThis is what it looks like:

Update image tags:
Image: mikefarah/yq:4
before_script:
# set the remote origin
- git remote set-url source "https://$REPO_TOKEN_NAME:$REPO_ACCESS_TOKEN@gitlab.com/myproject/myapp.git"
# configure git user
- git config --global user.email "ci@gitlab.com"
- git config --global user.name "GitLab CI"
# revert to remote master branch
- git fetch
- git master switch
- git restore -- hard origin/main
Scenario:
# update the YAML file
- yq -i ''$TAG_PATH' = "'$TAG'"' $FILE_PATH
-git append $FILE_PATH
- git commit -m "CI implementation of $TAG_PATH to $TAG"
- git push -o ci.skip origin HEAD:main

from the insideprevious scenarioIn the pipeline part, we created a Git repository and thisMrbranch (for older projects, you may need to adapt to master branch instead of master branch). In the script section we update the necessary YAML, create a new commit with the changes and push it to the source branch. we use-o ci. to skipto avoid triggering new commits through any other pipeline.

Scenario A: A repository

If you have a repository that contains your app code and Helm maps, your folder structure might look like this:

My application
source/
...
graph/
role model/
...
value.yaml
.gitlab-ci.yaml
conduit/
Update the images.yaml tags

In this case, we can place our pipeline templates in separate files and folders under the queue, e.g.pipeline/tags-images-updates.yaml.

In your existing build pipeline (.gitlab-ci.yaml), you can add a new stage for automatic deployment. This phase should be run after creating a new docker image and adding a specific tag. From development to development, this could be the SHORT SHA of the current commit that started the pipeline.

We will use GitLab CItriggerThe keyword used to locate and execute our template string. Variables are defined in the task and will be passed toimage tag updatechild labour.

phase:
- build_and_push
- development
Create a development task:
Faza: build_and_push
# ... build and push docker image with $CI_COMMIT_SHORT_SHA as tag
Implementation tasks:
Phase: implementation
Required: ["Create Development Task"]
trigger:
Include: cjevovodi/update-image-tag.yaml
variable:
FILE_PATH: charts/values.yaml
Flag: $CI_COMMIT_SHORT_SHA
TAG_PATH：.myapp.image.tag

Scenario B: Multiple repositories

The second case is a bit more complicated because the application and the Helm graphs are in multiple repositories. For this use case, we will leverage GitLabmulti-project pipeline.

Consider the following setup with two repositories:

My application
source/
...
.gitlab-ci.yaml

my map
graph/
role model/
...
value.yaml
.gitlab-ci.yaml
conduit/
Update the images.yaml tags

conductor forMy applicationThe repo looks like the scriptONE,but instead of usingincludetrigger, we will useWorktrigger. There you have to set the routemy mapwarehouse. Variables are defined in the same way as before.

phase:
- build_and_push
- development
Create a development task:
Faza: build_and_push
# ... build and push a docker image, e.g. marked $CI_COMMIT_SHORT_SHA
Implementation tasks:
Phase: implementation
Required: ["Create Development Task"]
trigger:
Project: My Projects/My Charts
Branch: main
variable:
FILE_PATH: charts/values.yaml
Flag: $CI_COMMIT_SHORT_SHA
TAG_PATH：.myapp.image.tag

from the insidemy mapIn the repo where the Helm diagram resides, we need to define a pipeline (.gitlab-ci.yaml) can only activate other items. That's what we'll useRulekeywords and environment variable controlCI_PIPELINE_SOURCE.Then we need to set the location of the pipeline templateimage tag updateTo work. All variables from the original CI tag rule will be passed through.

Implementation tasks:
Rule:
- if: $CI_PIPELINE_SOURCE == "pipeline"
trigger:
Include: cjevovodi/update-image-tag.yaml

Depending on your project, you may want to add more constraints when your pipeline is running. If you are not familiar with GitLab CI/CD, you can find a tutorialhere.

With this setup, you can now create fully automated pipelines for deploying GitOps setups in GitLab. The GitLab CI/CD developer experience still has room for improvement, but once set up properly, it's easily reusable.

Last but not least, if you find an easier way to achieve the same result, please let me know.