More Concise Pipeline Definition and Cross-Project Reuse¶
Gems and Jewels to Collect¶
In this episode we will discuss ways to take a CI pipeline apart and
extract parts of the pipeline into separate YAML files that can be included
into the main
.gitlab-ci.yml YAML file or even reused in other projects in
the same or different GitLab instances.
Finally, we will show you how to benefit from existing GitLab CI/CD Templates
and include these in your pipeline.
With time your CI pipeline might become large with several hundred lines of code and the pipeline might lack some structuring. You may split it up into multiple YAML files to give it a structure that is easier to handle and to arrange it more clearly. These YAML files can also be reused in different places, i.e. projects in the same or even in different GitLab instances.
include YAML files¶
used here is the keyword of your choice if you want to include YAML files
into another YAML file.
Imagine you have defined a YAML file that appears to be useful for many
Python projects: An example could be a YAML file
that is linting Python code and is put into directory
your Python project:
.prepare_linting: image: python:3.9 stage: lint before_script: - pip install --upgrade pip - pip install poetry - poetry install license_compliance: extends: .prepare_linting script: - poetry run reuse lint lint: extends: .prepare_linting script: - poetry run black --check --diff . - poetry run isort --check --diff .
It can be included into the main
.gitlab-ci.yml YAML file as follows:
As you can see the keyword
include has got sub-keys namely
template which we will explore in the next sections.
Include Files Inside the Same Project with
As demonstrated above, the simplest way to include YAML files in another one
is to put the file to include somewhere in the same project and reference it
in the main YAML file
.gitlab-ci.yml by using the relative
path from the project root to the YAML file to include.
Include Files from the Same GitLab Instance with
The extracted YAML file might not be contained in the same project but in
a different project in the same GitLab instance.
The corresponding reusable YAML given above can be referenced with the
Include Files from Other GitLab Instances with
If the reusable YAML file is not even located in the same GitLab instance,
we could make use of the YAML with the
Include GitLab CI/CD Templates with
At the time being the above-mentioned Python.gitlab-ci.yml template looks like this:
# To contribute improvements to CI/CD templates, please follow the Development guide at: # https://docs.gitlab.com/ee/development/cicd/templates.html # This specific template is located at: # https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Python.gitlab-ci.yml # Official language image. Look for the different tagged releases at: # https://hub.docker.com/r/library/python/tags/ image: python:latest # Change pip's cache directory to be inside the project directory since we can # only cache local items. variables: PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip" # Pip's cache doesn't store the python packages # https://pip.pypa.io/en/stable/topics/caching/ # # If you want to also cache the installed packages, you have to install # them in a virtualenv and cache it as well. cache: paths: - .cache/pip - venv/ before_script: - python --version # For debugging - pip install virtualenv - virtualenv venv - source venv/bin/activate test: script: - python setup.py test - pip install tox flake8 # you can also use tox - tox -e py36,flake8 run: script: - python setup.py bdist_wheel # an alternative approach is to install and run: - pip install dist/* # run the command here artifacts: paths: - dist/*.whl pages: script: - pip install sphinx sphinx-rtd-theme - cd doc - make html - mv build/html/ ../public/ artifacts: paths: - public rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
Split the Pipeline Definitions into Multiple Files¶
Now we are going to split our rather long
.gitlab-ci.yml YAML file into
multiple YAML files to:
- increase readability and maintainability,
- reduce duplication of the same configuration in multiple places,
- reuse complete YAML files in different projects of the same kind.
Let us first split it into three files, one for each stage:
.gitlab/ci/lint.gitlab-ci.ymldefines all CI jobs in stage
.gitlab/ci/test.gitlab-ci.ymldefines all CI jobs in stage
.gitlab/ci/run.gitlab-ci.ymldefines all CI jobs in stage
Now, to include all those YAML files in file
.gitlab-ci.yml, you can list
all files to include with the
This way your main CI-file stays clearly arranged, and it is easier to orient and get a good first overview. If you need to look deeper into the structure you will very easily spot the file in which you need to look into.
Make Use of GitLab CI/CD Templates¶
GitLab CI/CD Templates
of the GitLab project on GitLab.com offer a huge library of sophisticated
You can use an original or a modified version of a template and create a
YAML file to be included, e.g. with the
You can also include the file as is into your CI pipeline with the
include:template sub-key if it already works out of the box for your project.
Be aware that these templates might be subject to change over the course of
time if you consider to include templates from GitLab.
Exercise 1 - Restructure the CI Pipeline of the Exercise Project¶
In this final episode we used the
include keyword to split up a CI pipeline
into multiple files.
Now try to apply this way of restructuring on the current version of our
exercise CI pipeline.
Take Home Messages
Our last episode about mastering GitLab CI pipelines was about how to
.gitlab-ci.yml file even more by splitting it up into
several files that can be included in the main
How to Continue?¶
Your path to become a master in GitLab CI pipelines has been laid out. These resources will help you when you are about to develop CI pipelines in your own projects. In the next session of this workshop we will focus on your code projects and help you to get started with your CI projects. After that you will be able to dive deeper into the topic on your own by working on your specific requirements of your CI pipelines and reading more in the official GitLab documentation about concepts, keywords, and templates.