I’m frequently asked whether included CI/CD components can reference additional files stored outside of the pipeline repository. While including components in your configuration is straightforward since they’re just YAML, many users want to know if those included components can access and execute additional files referenced by the components, like shell scripts or other dependencies.
This challenge has been a common topic of discussion in threads across the GitLab Forum and Reddit.
Now for the good news: CI/CD components not only allow you to reuse pipeline configurations, saving time and effort, but you can also go a step further. With the new CI/CD Steps programming language, you can directly reuse centralized automation scripts and dependencies in your pipelines. You’ll gain even greater flexibility, making your pipelines more powerful and adaptable than ever.
By storing your scripts in a central location and wrapping them in CI/CD Steps, you can easily call these steps from your CI/CD components. This eliminates the need to duplicate scripts across multiple repositories and CI/CD configurations, streamlining your workflow and reducing redundancy.
Before we dive into the step-by-step guide, let’s briefly explore what CI/CD components and CI/CD Steps are.
What are CI/CD components?
CI/CD components are reusable units of pipeline configurations that get included in a pipeline when it’s created. The components bring additional jobs into the pipeline, however they can’t bring additional files as such reusable scripts.
What are CI/CD Steps?
CI/CD Steps are reusable units of a job. Each step defines structured inputs and outputs that can be consumed by other steps. Steps can come from local files, GitLab.com repositories, or any other Git source. Steps offer a structured alternative to shell scripts for running jobs. They are modular, can be composed, tested, and easily reused, providing greater flexibility and maintainability.
What are the differences between CI/CD Steps and CI/CD components?
-
Component and step definitions look very similar but they take effect at different phases in pipeline execution.
-
Components are used when a pipeline is created while steps are used when individual jobs are running.
-
When a step is running, the whole repository is being downloaded into the job environment along with extra files.
A step-by-step guide
Here is how CI/CD Steps and Components work together to access additional files.
This diagram illustrates the process flow: Jobs defined within components are imported into the pipeline configuration (.gitlab-ci.yml) when the pipeline is created. During the pipeline’s execution, a job’s steps are executed, and the entire Git repository is downloaded to the Step runner within the job’s context. This ensures that references to dependencies function correctly.
1. Define a component with run keyword that runs CI/CD Steps
Run is a new keyword that supports running steps, see the example code below. You can use this guide to learn more on how to create Components.
2. Create a step.yml file in the project where your scripts and dependencies are located.
In this code example, format.sh exists in the same directory as the step.yml.
While the job is running, the Step runner will download the entire Git repository where the step is defined. The ${{ step_dir }} step expression references the directory of the locally cached step files, allowing you to access other files from the repository. In the example above, the “format” step invokes the format.sh script.
3. Make sure that any files accessed by the step are located in the same repository as the step.yml file.
4. Include the component in your CI/CD configuration.
See this example code:
Code example: You can find the entire code demonstrated in this blog in this GitLab Group.
Important note: The CI/CD Steps feature is currently Experimental, and the syntax may change as we continue to iterate and refine it based on user feedback. Any feedback should be provided via this issue.
Learn more
-
Watch this walkthrough by Joe Burnett, principal engineer at GitLab, as he demonstrates the example discussed in the blog post.