Skip to content

More Concise Pipeline Definition and Cross-Project Reuse

Gems and Jewels to Collect

In this episode we will discuss ways to take a CI pipeline apart and extract parts of the pipeline into separate YAML files that can be included into the main .gitlab-ci.yml YAML file or even reused in other projects in the same or different GitLab instances. Finally, we will show you how to benefit from existing GitLab CI/CD Templates and include these in your pipeline.

Introduction

With time your CI pipeline might become large with several hundred lines of code and the pipeline might lack some structuring. You may split it up into multiple YAML files to give it a structure that is easier to handle and to arrange it more clearly. These YAML files can also be reused in different places, i.e. projects in the same or even in different GitLab instances.

include YAML files

The include keyword used here is the keyword of your choice if you want to include YAML files into another YAML file. Imagine you have defined a YAML file that appears to be useful for many Python projects: An example could be a YAML file lint.gitlab-ci.yml that is linting Python code and is put into directory .gitlab/ci/ of your Python project:

.prepare_linting:
  image: python:3.9
  stage: lint
  before_script:
    - pip install --upgrade pip
    - pip install poetry
    - poetry install

license_compliance:
  extends: .prepare_linting
  script:
    - poetry run reuse lint

lint:
  extends: .prepare_linting
  script:
    - poetry run black --check --diff .
    - poetry run isort --check --diff .

It can be included into the main .gitlab-ci.yml YAML file as follows:

stages:
  - lint

include:
  - local: '.gitlab/ci/lint.gitlab-ci.yml'

As you can see the keyword include has got sub-keys namely local, file, remote and template which we will explore in the next sections.

Include Files Inside the Same Project with include:local

As demonstrated above, the simplest way to include YAML files in another one is to put the file to include somewhere in the same project and reference it with the include:local sub-key in the main YAML file .gitlab-ci.yml by using the relative path from the project root to the YAML file to include.

Include Files from the Same GitLab Instance with include:file

The extracted YAML file might not be contained in the same project but in a different project in the same GitLab instance. The corresponding reusable YAML given above can be referenced with the include:file sub-key like this:

stages:
  - lint

include:
  - project: 'my-group/my-project'
    ref: 'main'
    file: '.gitlab/ci/lint.gitlab-ci.yml'

Include Files from Other GitLab Instances with include:remote

If the reusable YAML file is not even located in the same GitLab instance, we could make use of the YAML with the include:remote sub-key:

stages:
  - lint

include:
  - remote: 'https://gitlab.com/my-group/my-project/-/raw/main/.gitlab/ci/lint.gitlab-ci.yml'

Include GitLab CI/CD Templates with include:template

If your use-case is a common one, you could also use one of the GitLab CI/CD Templates of the GitLab project on GitLab.com with the include:template sub-key:

include:
  - template: 'Python.gitlab-ci.yml'

At the time being the above-mentioned Python.gitlab-ci.yml template looks like this:

# To contribute improvements to CI/CD templates, please follow the Development guide at:
# https://docs.gitlab.com/ee/development/cicd/templates.html
# This specific template is located at:
# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Python.gitlab-ci.yml

# Official language image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/python/tags/
image: python:latest

# Change pip's cache directory to be inside the project directory since we can
# only cache local items.
variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

# Pip's cache doesn't store the python packages
# https://pip.pypa.io/en/stable/topics/caching/
#
# If you want to also cache the installed packages, you have to install
# them in a virtualenv and cache it as well.
cache:
  paths:
    - .cache/pip
    - venv/

before_script:
  - python --version  # For debugging
  - pip install virtualenv
  - virtualenv venv
  - source venv/bin/activate

test:
  script:
    - python setup.py test
    - pip install tox flake8  # you can also use tox
    - tox -e py36,flake8

run:
  script:
    - python setup.py bdist_wheel
    # an alternative approach is to install and run:
    - pip install dist/*
    # run the command here
  artifacts:
    paths:
      - dist/*.whl

pages:
  script:
    - pip install sphinx sphinx-rtd-theme
    - cd doc
    - make html
    - mv build/html/ ../public/
  artifacts:
    paths:
      - public
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Split the Pipeline Definitions into Multiple Files

Now we are going to split our rather long .gitlab-ci.yml YAML file into multiple YAML files to:

  • increase readability and maintainability,
  • reduce duplication of the same configuration in multiple places,
  • reuse complete YAML files in different projects of the same kind.

Let us first split it into three files, one for each stage:

  1. File .gitlab/ci/lint.gitlab-ci.yml defines all CI jobs in stage lint:
    license_compliance:
      stage: lint
      script:
        - poetry run reuse lint
    
    lint:
      stage: lint
      script:
        - poetry run black --check --diff .
        - poetry run isort --check --diff .
    
  2. File .gitlab/ci/test.gitlab-ci.yml defines all CI jobs in stage test:
    test:python:
      image: python:${VERSION}
      stage: test
      script:
        - poetry run pytest tests/
      parallel:
        matrix:
          - VERSION: ["3.8", "3.9", "3.10"]
      needs: ["lint"]
    
  3. File .gitlab/ci/run.gitlab-ci.yml defines all CI jobs in stage run:
    run:
      stage: run
      script:
        - poetry run python -m astronaut_analysis
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      artifacts:
        paths:
          - results/
      needs: ["test:python: [3.9]"]
    

Now, to include all those YAML files in file .gitlab-ci.yml, you can list all files to include with the include:local sub-key:

stages:
  - lint
  - test
  - run

default:
  image: python:3.9
  before_script:
    - pip install --upgrade pip
    - pip install poetry
    - poetry install

include:
  - local: '.gitlab/ci/lint.gitlab-ci.yml'
  - local: '.gitlab/ci/test.gitlab-ci.yml'
  - local: '.gitlab/ci/run.gitlab-ci.yml'

This way your main CI-file stays clearly arranged, and it is easier to orient and get a good first overview. If you need to look deeper into the structure you will very easily spot the file in which you need to look into.

Make Use of GitLab CI/CD Templates

The GitLab CI/CD Templates of the GitLab project on GitLab.com offer a huge library of sophisticated reference templates. You can use an original or a modified version of a template and create a YAML file to be included, e.g. with the include:local sub-key. You can also include the file as is into your CI pipeline with the include:template sub-key if it already works out of the box for your project. Be aware that these templates might be subject to change over the course of time if you consider to include templates from GitLab.

Exercise

Exercise 1 - Restructure the CI Pipeline of the Exercise Project

In this final episode we used the include keyword to split up a CI pipeline into multiple files. Now try to apply this way of restructuring on the current version of our exercise CI pipeline.

Take Home Messages

Our last episode about mastering GitLab CI pipelines was about how to structure the .gitlab-ci.yml file even more by splitting it up into several files that can be included in the main .gitlab-ci.yml file.

How to Continue?

Your path to become a master in GitLab CI pipelines has been laid out. These resources will help you when you are about to develop CI pipelines in your own projects. In the next session of this workshop we will focus on your code projects and help you to get started with your CI projects. After that you will be able to dive deeper into the topic on your own by working on your specific requirements of your CI pipelines and reading more in the official GitLab documentation about concepts, keywords, and templates.