Variables inside Azure Pipelines have a long history of complexity. I get the feeling that they started off pretty simple and primitive. Over time the introduction of new features like YAML pipelines along with templating and expressions have made a capable variable syntax that can be used in a lot of situations. I’d argue, however, that as a result, it’s a somewhat disparate, inconsistent, and dense experience. It is not just about user-defined, system-defined, or environment-defined variables. There are multiple levels of precedence and priority of usage, predefined variables, variable groups, and release variables. But my favorite complexity in Azure Pipelines Variables is working through YAML stages and using variables as it relates to macro, templates, and runtime syntax. If we pull this together with dynamic variables created in scripts, it’s possible to compose some pretty advanced pipeline workflows with relatively little code. To do this, you’ll additionally need to manage how to reference variables between jobs and stages which I’m still a bit confused about in certain scenarios.
Variables are quite easy to define in script output. You will want to pay attention to the modifications on variable creation, especially around secrets and output variables. Reasons for specifying a variable as a secret are obvious, and it strips the value from visibility in the log output. An output variable though is necessary if you intend to use the variable outside of the current job, in another job or stage, that is ultimately executed on a different agent or host entirely (most likely).
- script: echo "##vso[task.setvariable variable=buildJobVar;isOutput=true]fromBuildStage" name: buildJobStep
It is critical for your script that is outputting the dynamic variable to specify a name for the step. This will be used to access the variable to reduce the possibility of naming collisions.
The variable can be accessed in future jobs or deployment jobs by specifying the dependency in the variables of the job. The convention used should be obvious in the following example referencing dependencies in the stage by name, the job name, followed by the outputs of a particular step and variable from that step (implying you could have multiple variables output from a single step). It’s also worth noting that the stage you access the variable must have the source variable stage as a dependency, either implicitly or explicitly (in the example below no dependsOn is specified so it is an implicit dependsOn for the previous stage by default). You must use the runtime syntax in this case to specify the variable in the variables of the job to proxy its access through the various levels of YAML rendering (i.e. you cannot access the value of stageDependencies as a runtime syntax – or macro syntax – for a task within the job directly).
- stage: buildStage jobs: - job: buildJob pool: YOUR-BUILD steps: - script: echo "##vso[task.setvariable variable=buildJobVar;isOutput=true]fromBuildStage" name: buildJobStep - stage: integrationStage jobs: - deployment: integrationJob pool: YOUR-BUILD environment: nonprod variables: myBuildStageVar: $[stageDependencies.buildStage.buildJob.outputs['buildJobStep.buildJobVar']] strategy: runOnce: deploy: steps: - script: echo $(myBuildStageVar) name: buildStageVar
Thus far, the above examples are pretty straightforward and outlined pretty well in the documentation, though it could use a few more examples and explanations. However, if we add in further workflow and changes to automatically skip stages based on prior stage runtime variables and then review the differences in doing so between regular job types and deployment jobs, it definitely demonstrates some inconsistencies.
The easiest way to visualize this I think is through an example that outlines 3 different jobs. The first job is a regular job type that might typically be for a “build” or compile stage. The second stage has a deployment job, which might be like your “dev” or “integration” deployment stage. This stage depends on the build job variable before it. The third job is similar to a “prod” deployment stage that depends on a variable condition from the “integration” deployment stage.
Note the naming convention of the stages and jobs, as it is intended to make referencing the hierarchy of stages, jobs, and tasks a bit easier in the YAML below.
trigger: none pr: none stages: - stage: buildStage jobs: - job: buildJob pool: YOUR-POOL steps: - script: echo "##vso[task.setvariable variable=buildJobVar;isOutput=true]fromBuildStage" name: buildJobStep - stage: integrationStage condition: eq(dependencies.buildStage.outputs['buildJob.buildJobStep.buildJobVar'], 'fromBuildStage') jobs: - deployment: integrationJob pool: YOUR-POOL environment: integration variables: myBuildStageVar: $[stageDependencies.buildStage.buildJob.outputs['buildJobStep.buildJobVar']] strategy: runOnce: deploy: steps: - script: echo $(myBuildStageVar) name: buildStageVar - script: echo "##vso[task.setvariable variable=deployJobVar;isOutput=true]fromIntegrationStage" name: deployJobStep - stage: prodStage condition: eq(dependencies.integrationStage.outputs['integrationJob.integrationJob.deployJobStep.deployJobVar'], 'fromIntegrationStage') dependsOn: - buildStage - integrationStage jobs: - deployment: prodJob pool: YOUR-POOL environment: prod variables: myBuildStageVar: $[stageDependencies.buildStage.buildJob.outputs['buildJobStep.buildJobVar']] myDeployStageVar: $[stageDependencies.integrationStage.integrationJob.outputs['integrationJob.deployJobStep.deployJobVar']] strategy: runOnce: deploy: steps: - script: echo $(myBuildStageVar) name: buildStageVar - script: echo $(myDeployStageVar) name: deployStageVar
Using the above example reference you should be able to effectively build multiple stages that execute based on conditions using prior stage dependencies and variables. That being said, here are some VERY NOTABLE observations:
build_jobappears twice. I cannot tell how this conventionally makes sense. Without that single line in the documentation that has no explanation, I would have assumed deployment job references were utterly broken. The use of the dual Job Name is also needed in variable references to other deployment jobs as well using stageDependencies.
Definitely some odd conventions with variables between stages. Even still, I cannot wrap my head around entirely remembering them when I need them. This example pipeline is handy as a reference when you need to remember how to use and access variables in different stages based on different job types.
UPDATE: After some helpful support from the Microsoft team, the above information has been updated to indicate the need for the “dependsOn” property for secondary references, and the usage of dual job names specified in variable references.