Viotex

Introduction

It is part of our philosophy, that we strive to automate every aspect of the job, that is worth automating and will leave us more time to spend in creative and exploratory activities. Since we use Gitlab as main DevOps platform we make sure to use its CI capabilities. Here is a practical example of such use case, that might be useful to others.

GitLab CI/CD pipelines provide a powerful mechanism for automating and orchestrating various stages of software development, from building and testing to deployment. However, when working with multi-project pipelines and downstream dependencies, certain best practices should be considered to avoid common pitfalls. Below are some key considerations (written in blood):

Best Practices

1. Global Variables

• Issue: Global variables defined in the upstream pipeline will overwrite downstream global variables.
• Consideration: Variables set at the global level in the upstream pipeline are accessible by downstream pipelines, potentially overwriting their values.
• Best Practice: Use job-level variables or conditionally set global variables based on specific rules to avoid unintentional overwriting.

Let’s illustrate the scenario with a global variable example:

# Upstream pipeline (in project: upstream_project)
variables:
  DOCKER_CORE_IMAGE_URI: "$CI_REGISTRY/core_project/image:dev"

upstream_job:
  script:
    - echo "Building and generating artifacts..."
    # Your build and generation commands here
  artifacts:
    paths:
      - path/to/artifact   # Specify the path to the artifact you want to pass downstream

Now, let’s consider the downstream pipeline:

# Downstream pipeline (in project: downstream_project)
variables:
  DOCKER_CORE_IMAGE_URI: "$CI_REGISTRY/core_project/image"

downstream_job:
  script:
    - echo "Running downstream job..."
    - echo "Using the downstream Docker image URI: $DOCKER_CORE_IMAGE_URI"
    # Your downstream job commands here
  needs:
    - project: upstream_project
      job: upstream_job
      ref: main  # Specify the branch or commit ref of the upstream job
      artifacts: true

The global variable $DOCKER_CORE_IMAGE_URI‘s value from the upstream pipeline will be used in the downstream pipeline!

You could disable global variables inheritance explicitly it in the upstream pipeline, so that downstream won’t receive any upstream global variables:

upstream_job:
  ...
  inherit:
    variables: false # Disable the inheritance of all global variables
  trigger: # Define the details for triggering downstream pipeline
    ...
    strategy: depend # This job depends on the success of the triggered pipeline

2. Access Control for Downstream

3. Check if Pipeline has been Triggered

If you want to determine whether a job has been triggered as part of a downstream pipeline, you can check the predefined environment variable $CI_PIPELINE_SOURCE. The value will indicate the source of the pipeline.

your_job:
  ...
  rules:
    - if: $CI_PIPELINE_SOURCE == "pipeline"
  ...

4. Passing Downstream Artifacts

Suppose you have an upstream pipeline with the following job:

# Upstream pipeline (in project: upstream_project)
upstream_job:
  script:
    - echo "Building and generating artifacts..."
    # Your build and generation commands here
  artifacts:
    paths:
      - path/to/artifact   # Specify the path to the artifact you want to pass downstream

Now, you have a downstream pipeline in another project downstream_project that needs the artifacts from the upstream job. You can express this dependency using the needs keyword:

# Downstream pipeline (in project: downstream_project)
downstream_job:
  script:
    - echo "Running downstream job..."
    # Your downstream job commands here
  needs:
    - project: upstream_project
      job: upstream_job
      ref: main  # Specify the branch or commit ref of the upstream job
      artifacts: true

5. Important assumptions

The above examples were implemented based on Gitlab version 15.x and still operates on 16.x version premium setup we use. Gitlab are constantly trying to improve their CI functionality, and specifics might change. But in our opinion Gitlab CI is definitely their top competitive advantage we believe stuff can only get better :).

6. Feedback

If you have questions on this or similar use cases regarding Gitlab CI or general CI/CD workflow, practices and procedures, we are always open to answer questions or help. So don’t hesitate to write us!