Annotate container images with build provenance using Cosign in GitLab CI/CD

Container security has become a critical concern in software development. As organizations increasingly rely on containerized applications, ensuring the integrity and traceability of container images is paramount. Enhancing the security and traceability of your container images directly in your GitLab CI/CD pipeline can streamline your development process while significantly boosting your security posture.

This tutorial demonstrates setting up a GitLab pipeline to automate the process of building, signing, and annotating Docker images using Cosign and the GitLab container registry. By integrating these practices, you’ll secure your images and ensure that each one is easily traceable, aligning with best practices in DevSecOps.

Background on container image security

Before we dive into the technical details, it’s crucial to understand why container image security is so important. In microservices and cloud-native applications, containers have become the standard for packaging and deploying software. However, this widespread adoption has also made containers an attractive target for cyber attacks.

Container image security is a vital component of the broader software supply chain security concept. This encompasses all the tools, processes, and practices that ensure your software’s integrity, authenticity, and security from development to deployment. By securing your container images, you’re protecting your application and your entire software supply chain.

Introduction to Cosign

Enter Cosign, a tool designed to address these security concerns. Cosign is part of the Sigstore project, an open-source initiative aimed at improving the security of the software supply chain. Cosign allows developers to sign and verify container images, ensuring their integrity and authenticity.

Key benefits of Cosign include:

easy integration with existing CI/CD pipelines
support for various signing methods, including keyless signing
ability to attach and verify arbitrary metadata to container images

By incorporating Cosign into your GitLab CI/CD pipeline, you’re taking a significant step towards robust DevSecOps practices.

Benefits of image signing and annotation

Image signing serves as a seal of authenticity for your container images. It helps prevent tampering and ensures that the image deployed in your production environment is precisely the one that passed through your secure build process.

Annotations, on the other hand, provide valuable metadata about the build process. This information is used for auditing and traceability. In a security incident, having detailed provenance data can significantly speed up the investigation and remediation process.

GitLab CI/CD pipeline configuration

Let’s look at an example .gitlab-ci.yml file that outlines the process of building, signing, and annotating a Docker image using Cosign:

stages:
  - build

build_and_sign:
  stage: build
  image: docker:latest
  services:
    - docker:dind  # Enable Docker-in-Docker service to allow Docker commands inside the container
  variables:
    IMAGE_TAG: $CI_COMMIT_SHORT_SHA  # Use the commit short SHA as the image tag
    IMAGE_URI: $CI_REGISTRY_IMAGE:$IMAGE_TAG  # Construct the full image URI with the registry, project path, and tag
    COSIGN_YES: "true"  # Automatically confirm actions in Cosign without user interaction
    FF_SCRIPT_SECTIONS: "true"  # Enables GitLab's CI script sections for better multi-line script output
  id_tokens:
    SIGSTORE_ID_TOKEN:
      aud: sigstore  # Provide an OIDC token for keyless signing with Cosign
  before_script:
    - apk add --no-cache cosign jq  # Install Cosign (mandatory) and jq (optional)
    - docker login -u "gitlab-ci-token" -p "$CI_JOB_TOKEN" "$CI_REGISTRY"  # Log in to the Docker registry using GitLab CI token
  script:
    # Build the Docker image using the specified tag and push it to the registry
    - docker build --pull -t "$IMAGE_URI" .
    - docker push "$IMAGE_URI"

    # Retrieve the digest of the pushed image to use in the signing step
    - IMAGE_DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE_URI")

    # Sign the image using Cosign with annotations that provide metadata about the build and tag annotation to allow verifying
    # the tag->digest mapping (https://github.com/sigstore/cosign?tab=readme-ov-file#tag-signing)
    - |
      cosign sign "$IMAGE_DIGEST" 
        --annotations "com.gitlab.ci.user.name=$GITLAB_USER_NAME" 
        --annotations "com.gitlab.ci.pipeline.id=$CI_PIPELINE_ID" 
        --annotations "com.gitlab.ci.pipeline.url=$CI_PIPELINE_URL" 
        --annotations "com.gitlab.ci.job.id=$CI_JOB_ID" 
        --annotations "com.gitlab.ci.job.url=$CI_JOB_URL" 
        --annotations "com.gitlab.ci.commit.sha=$CI_COMMIT_SHA" 
        --annotations "com.gitlab.ci.commit.ref.name=$CI_COMMIT_REF_NAME" 
        --annotations "com.gitlab.ci.project.path=$CI_PROJECT_PATH" 
        --annotations "org.opencontainers.image.source=$CI_PROJECT_URL" 
        --annotations "org.opencontainers.image.revision=$CI_COMMIT_SHA" 
        --annotations "tag=$IMAGE_TAG"

    # Verify the image signature using Cosign to ensure it matches the expected annotations and certificate identity
    - |
      cosign verify 
        --annotations "tag=$IMAGE_TAG" 
        --certificate-identity "$CI_PROJECT_URL//.gitlab-ci.yml@refs/heads/$CI_COMMIT_REF_NAME" 
        --certificate-oidc-issuer "$CI_SERVER_URL" 
        "$IMAGE_URI" | jq .  # Use jq to format the verification output for easier readability

Let’s break down this pipeline configuration and understand each part in detail.

Detailed explanation of the pipeline

1. Setup and prerequisites

The pipeline starts by setting up the necessary environment:

It uses the docker:latest image and enables Docker-in-Docker service, allowing Docker commands to be run within the CI job.
It defines variables for the image tag and URI using GitLab CI/CD predefined variables.
It sets up an OIDC token for keyless signing with Cosign.
In the before_script section, it installs Cosign and jq (for JSON processing) and logs into the GitLab container registry.

2. Building and pushing the image

The first step in the script is to build the Docker image and push it to the GitLab container registry:

- docker build --pull -t "$IMAGE_URI" .
- docker push "$IMAGE_URI"

This creates the image using the current directory’s Dockerfile and pushes it to the registry.

3. Signing the image with Cosign

After building and pushing the image, the pipeline signs it using Cosign:

- IMAGE_DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE_URI")
- |
  cosign sign "$IMAGE_DIGEST" 
    --annotations "com.gitlab.ci.user.name=$GITLAB_USER_NAME" 
    --annotations "com.gitlab.ci.pipeline.id=$CI_PIPELINE_ID" 
    # ... (other annotations) ...
    --annotations "tag=$IMAGE_TAG"

This step first retrieves the image digest and then uses Cosign to sign the image, adding several annotations.

Verifying the signature and annotations

After signing the image, it’s crucial to verify the signature and the annotations we’ve added. This verification step ensures that the provenance data attached to the image is correct and hasn’t been tampered with.

In our pipeline, we’ve included a verification step using the cosign verify command:

- |
  cosign verify 
    --annotations "tag=$IMAGE_TAG" 
    --certificate-identity "$CI_PROJECT_URL//.gitlab-ci.yml@refs/heads/$CI_COMMIT_REF_NAME" 
    --certificate-oidc-issuer "$CI_SERVER_URL" 
    "$IMAGE_URI" | jq .

This command verifies the signature and checks the annotations. Its output will show all the annotations we’ve added to the image during the signing process.

Here’s what you might see in your pipeline logs after running this command:

verifying the signature and checking annotations

In this output, you should see all the annotations we added earlier, including:

GitLab CI user name
Pipeline ID and URL
Job ID and URL
Commit SHA and reference name
Project path
Image source and revision

By verifying these annotations, you can ensure that the image’s provenance data is intact and matches what you expect based on your build process. This verification step is crucial for maintaining the integrity of your software supply chain. It allows you to confirm that the image you’re about to deploy has gone through your secure build process and has yet to be modified since it was signed.

Summary

By integrating Cosign into your GitLab CI/CD pipeline, you’ve taken a significant step toward securing your software supply chain. This setup not only automates securing and annotating your container images with build metadata but also ensures a transparent and traceable build process.

The benefits of this approach are numerous:

enhanced security through image signing
improved traceability with detailed build provenance data
automated verification process
alignment with DevSecOps best practices

As container security continues to be a critical concern in the software development lifecycle, implementing these practices puts you ahead of potential security threats and demonstrates a commitment to software integrity.

Try it in your organization

Now that you’ve seen how to enhance your container security using Cosign in GitLab CI/CD, it’s time to put this knowledge into practice:

Implement in your projects: Adapt the provided .gitlab-ci.yml file to fit your specific needs.
Explore further: Dive deeper into Cosign’s capabilities. Consider exploring advanced features like policy enforcement or integration with vulnerability scanning tools.
Share your experience: After implementing this in your projects, share your experience with your team or the wider GitLab community. Your insights could help others enhance their security practices.
Stay updated: Container security is an evolving field. Check GitLab’s blog and documentation for new features and best practices updates.
Contribute: If you find ways to improve this process or encounter any issues, consider contributing to the GitLab or Cosign open-source projects.

Remember, security is a journey, not a destination. By taking these steps, you’re securing your containers and contributing to a more secure software ecosystem for everyone.

Start implementing these practices in your GitLab projects today, and take your container security to the next level!

Get started today! Sign up for a free 30-day trial of GitLab Ultimate!