Working with GitLab CI CD pipelines is a great way to automate your DevOps processes. In building a CI CD pipeline you can use variables effectively in the CI CD pipeline to reference values you have stored in your GitLab CI CD settings. It is a great way to handle secrets and have these referenced without hardcoding things in your code. Below I will show you what they are and also demonstrate a real-world example of how you can use them in GitLab.
Why use GitLab CI CD variables?
Hard coding values in your code for passwords or tokens is a very bad idea. Avoid hard coding values as it can lead to exposing extremely sensitive data or having your secrets fall into the wrong hands.
Secrets information should always be stored in the settings UI for a pipeline in GitLab and never in the .gitlab-ci.yml file.
- For a project in the project’s settings
- For all projects in a group in the group’s setting
- For all projects in a GitLab instance in the instance’s settings
Caveats to using the GitLab CI CD variables
One thing to remember is that forked projects can’t access the CI CD variables that are available in the parent project. However, if you run a merge request pipeline in the parent project for a merge request from a fork, then the variables will be available in the pipeline.
Protected, masked, and expanded
Even though you are using variables for sensitive data, information can still be exposed in a job log or sent to a third part server. GitLab CI CD has a masked variable feature that helps you reduce the likelihood of accidentally exposing variable values.
GitLab allows having several types of attributes:
- Protected variables: These are only exposed to protected branches or protected tags. For the most part, you may have non sensitive project configuration, but the protected variables are for sensitive projects.
- Masked variables: These are hidden in job logs
- Expanded: Expanded variables treat values as the start of a reference to another variable. Otherwise, the value will be treated as the start of a reference to another variable like variable type variables and what you would expect.
Globally defined variables
Normally for each CI CD pipeline, these are environment scoped project variables. You can also use globally defined variables on GitLab groups so projects in the same group can access environment variables set at this level and variables defined globally. The group variables can save work if you have many projects that are using the same variables. Keep the environment scope in mind when creating variables.
Creating GitLab CI CD pipeline variables
To create a GitLab CI CD pipeline variables, navigate to Settings > CI/CD > Variables > Expand > Add variable.
You can see variables that I have created for a pipeline that runs in my home lab that backs up VMware ESXi configuration to my NAS device. I am using the CI/CD variables to store the vCenter Server credentials and the NAS credentials, along with the vCenter Server hostname. Below, you can see the variables defined for my specific pipeline.
You can click the pencil icon next to each to see the environment variable value. These are simple key and value pair variables defined for sensitive variables.
Create a gitlab ci cd pipeline instance variable or environment variable for use with your Pipeline. Here you can see the Add variable dialog box. This is where you make selections for visibility, flags, description, key and other configuration.
How do you use GitLab CI CD pipeline variables?
So, how do you use the GitLab CI CD pipeline variables created in the GitLab UI settings for CI/CD? This is actually pretty easy to do. In my PowerCLI script in the above screenshot, you see the variables that will be used for my code to pull the vCenter Server, NAS user and pass and vSphere credentials.
Below is the snippet of code at the top that allows you to use the pipeline variables. As you can see, we use the $env:<VARIABLENAME> to reference the GitLab CI CD variable you configured. These are environment variables that are called from the GitLab variables stored for CI CD purposes.
# Define variables
$vcenterhost = $env:VCENTERHOST
$vsphereuser = $env:VSPHEREUSER
$vspherepassword = $env:VSPHEREPASSWORD
$nasuser = $env:NASUSER
$naspass = $env:NASPASS
Full PowerCLI code using the GitLab variables, so you can see this in context:
# Define variables
$vcenterhost = $env:VCENTERHOST
$vsphereuser = $env:VSPHEREUSER
$vspherepassword = $env:VSPHEREPASSWORD
$nasuser = $env:NASUSER
$naspass = $env:NASPASS
# Generate a timestamp for the parent folder
$timestamp = Get-Date -Format "yyyyMMdd_HHmmss"
$parentFolderPath = "./ESXiConfigs/$timestamp"
# Create the parent folder if it does not exist
if (-not (Test-Path -Path $parentFolderPath)) {
New-Item -ItemType Directory -Path $parentFolderPath -Force
}
## Setting CEIP options
Set-PowerCliConfiguration -InvalidCertificateAction Ignore -ParticipateInCeip $false -Confirm:$false
# Connect to vCenter Server
Connect-VIServer -Server $vcenterhost -User $vsphereuser -Password $vspherepassword
# Create the parent folder on the NAS
smbclient //10.1.149.4/esxiconfigs -U $nasuser --password $naspass -c "mkdir $timestamp"
# Get all ESXi hosts
$allHosts = Get-VMHost
foreach ($esxihost in $allHosts) {
# Define export paths
$hostFolderPath = "$parentFolderPath/$($esxihost.Name)"
$exportPath = "$hostFolderPath/$($esxihost.Name)_config.vSphereConfiguration"
$infoPath = "$hostFolderPath/$($esxihost.Name)_info.txt"
# Create the host-specific folder if it does not exist
if (-not (Test-Path -Path $hostFolderPath)) {
New-Item -ItemType Directory -Path $hostFolderPath -Force
}
# Export ESXi configuration to a file
# Create a temporary folder to hold the backup file as Get-VMHostFirmware requires a directory path
$tempFolder = "$hostFolderPath/temp"
if (-not (Test-Path -Path $tempFolder)) {
New-Item -ItemType Directory -Path $tempFolder -Force
}
Get-VMHostFirmware -VMHost $esxihost -BackupConfiguration -DestinationPath $tempFolder
# Move the backup file to the desired location
$backupFile = Get-ChildItem -Path $tempFolder -Filter *.tgz | Select-Object -First 1
if ($backupFile) {
Move-Item -Path $backupFile.FullName -Destination $exportPath
}
# Remove the temporary folder
Remove-Item -Path $tempFolder -Recurse
# Get ESXi version, build, and image profile information
$hostVersion = $esxihost.Version
$hostBuild = $esxihost.Build
$imageProfileName = $esxihost.ExtensionData.Config.ImageProfileName
$fullVersion = $esxihost.ExtensionData.Config.Product.FullName
# Create a text file with version, build, and image profile information
$infoContent = "ESXi Host: $($esxihost.Name)`n"
$infoContent += "Full Version: $fullVersion`n"
$infoContent += "Version: $hostVersion`n"
$infoContent += "Build: $hostBuild`n"
$infoContent += "Image Profile: $imageProfileName`n"
$infoContent | Out-File -FilePath $infoPath
# Upload files to the NAS using smbclient
smbclient //10.1.149.4/esxiconfigs -U $nasuser --password $naspass -c "cd $timestamp; put $exportPath $(basename $exportPath)"
smbclient //10.1.149.4/esxiconfigs -U $nasuser --password $naspass -c "cd $timestamp; put $infoPath $(basename $infoPath)"
}
# Disconnect from vCenter Server
Disconnect-VIServer -Server $vcenterhost -Confirm:$false
Write-Host "Export completed. Check $parentFolderPath for the configuration and info files."
Masked variables in job logs
What do masked GitLab CI CD pipeline variables look like in the job logs? As you can see below in the job log output, I have the vCenter Server as a masked variable, so we see it shown as [MASKED] in the logs or temporary file creations.
Troubleshooting
Let’s look at some troubleshooting in case the variables are not working from GitLab. If the variables are not working:
- Check to see you have referenced exactly the name fo the variable as it is configured in GitLab – It is easy for typos to creep in and cause issues with the GitLab pipelines
- Make sure the variable contains the right value – Check to make sure your variable contains the right value.
- If you see a pipeline variables error message, make note of the error and troubleshoot accordingly by looking at the pipeline logs and the CI CD instance variable being referred to.
Takeaways
Do use GitLab CI CD variables as they can help you avoid hard coding values in your code checked into the repo. Understand the scope of the variables defined and use them accordingly as instance variables or global.
