Azure Pipelines: Passing Variables Between Stages

Variables inside Azure Pipelines have a long history of complexity. I get the feeling that they started off pretty simple and primitive. Over time the introduction of new features like YAML pipelines along with templating and expressions have made a capable variable syntax that can be used in a lot of situations. I’d argue, however, that as a result, it’s a somewhat disparate, inconsistent, and dense experience. It is not just about user-defined, system-defined, or environment-defined variables. There are multiple levels of precedence and priority of usage, predefined variables, variable groups, and release variables. But my favorite complexity in Azure Pipelines Variables is working through YAML stages and using variables as it relates to macro, templates, and runtime syntax. If we pull this together with dynamic variables created in scripts, it’s possible to compose some pretty advanced pipeline workflows with relatively little code. To do this, you’ll additionally need to manage how to reference variables between jobs and stages which I’m still a bit confused about in certain scenarios.

Defining a Script Variable

Variables are quite easy to define in script output. You will want to pay attention to the modifications on variable creation, especially around secrets and output variables. Reasons for specifying a variable as a secret are obvious, and it strips the value from visibility in the log output. An output variable though is necessary if you intend to use the variable outside of the current job, in another job or stage, that is ultimately executed on a different agent or host entirely (most likely).

- script: echo "##vso[task.setvariable variable=buildJobVar;isOutput=true]fromBuildStage"
  name: buildJobStep

It is critical for your script that is outputting the dynamic variable to specify a name for the step. This will be used to access the variable to reduce the possibility of naming collisions.

Accessing a Variable from a Prior Job

The variable can be accessed in future jobs or deployment jobs by specifying the dependency in the variables of the job. The convention used should be obvious in the following example referencing dependencies in the stage by name, the job name, followed by the outputs of a particular step and variable from that step (implying you could have multiple variables output from a single step). It’s also worth noting that the stage you access the variable must have the source variable stage as a dependency, either implicitly or explicitly (in the example below no dependsOn is specified so it is an implicit dependsOn for the previous stage by default). You must use the runtime syntax in this case to specify the variable in the variables of the job to proxy its access through the various levels of YAML rendering (i.e. you cannot access the value of stageDependencies as a runtime syntax – or macro syntax – for a task within the job directly).

- stage: buildStage
  jobs:
  - job: buildJob
    pool: YOUR-BUILD
    steps:
    - script: echo "##vso[task.setvariable variable=buildJobVar;isOutput=true]fromBuildStage"
      name: buildJobStep 

- stage: integrationStage
  jobs:
  - deployment: integrationJob
    pool: YOUR-BUILD
    environment: nonprod
    variables:
      myBuildStageVar: $[stageDependencies.buildStage.buildJob.outputs['buildJobStep.buildJobVar']]
    strategy:
      runOnce:
        deploy:
          steps:
          - script: echo $(myBuildStageVar)
            name: buildStageVar

Conditions, Expressions, and Deployment Jobs

Thus far, the above examples are pretty straightforward and outlined pretty well in the documentation, though it could use a few more examples and explanations. However, if we add in further workflow and changes to automatically skip stages based on prior stage runtime variables and then review the differences in doing so between regular job types and deployment jobs, it definitely demonstrates some inconsistencies.

The easiest way to visualize this I think is through an example that outlines 3 different jobs. The first job is a regular job type that might typically be for a “build” or compile stage. The second stage has a deployment job, which might be like your “dev” or “integration” deployment stage. This stage depends on the build job variable before it. The third job is similar to a “prod” deployment stage that depends on a variable condition from the “integration” deployment stage.

Note the naming convention of the stages and jobs, as it is intended to make referencing the hierarchy of stages, jobs, and tasks a bit easier in the YAML below.

trigger: none
pr: none

stages:
- stage: buildStage
  jobs:
  - job: buildJob
    pool: YOUR-POOL
    steps:
    - script: echo "##vso[task.setvariable variable=buildJobVar;isOutput=true]fromBuildStage"
      name: buildJobStep 

- stage: integrationStage
  condition: eq(dependencies.buildStage.outputs['buildJob.buildJobStep.buildJobVar'], 'fromBuildStage')
  jobs:
  - deployment: integrationJob
    pool: YOUR-POOL
    environment: integration
    variables:
      myBuildStageVar: $[stageDependencies.buildStage.buildJob.outputs['buildJobStep.buildJobVar']]
    strategy:
      runOnce:
        deploy:
          steps:
          - script: echo $(myBuildStageVar)
            name: buildStageVar
          - script: echo "##vso[task.setvariable variable=deployJobVar;isOutput=true]fromIntegrationStage"
            name: deployJobStep

- stage: prodStage
  condition: eq(dependencies.integrationStage.outputs['integrationJob.integrationJob.deployJobStep.deployJobVar'], 'fromIntegrationStage')
  dependsOn:
  - buildStage
  - integrationStage
  jobs:
  - deployment: prodJob
    pool: YOUR-POOL
    environment: prod
    variables:
      myBuildStageVar: $[stageDependencies.buildStage.buildJob.outputs['buildJobStep.buildJobVar']]
      myDeployStageVar: $[stageDependencies.integrationStage.integrationJob.outputs['integrationJob.deployJobStep.deployJobVar']]
    strategy:
      runOnce:
        deploy:
          steps:
          - script: echo $(myBuildStageVar)
            name: buildStageVar
          - script: echo $(myDeployStageVar) 
            name: deployStageVar

Using the above example reference you should be able to effectively build multiple stages that execute based on conditions using prior stage dependencies and variables. That being said, here are some VERY NOTABLE observations:

  • stageDependencies vs dependencies – Depending on if you are referencing the prior stage variables in the conditions of the job or within the variables of the job requires two different root variables.
  • Job Name Reference Location – If specifying variables with dependencies in conditions, then the job name is referenced inside the outputs compared before the outputs (dependencies.buildStage.outputs… vs stageDependencies.buildStage.buildJob.outputs).
  • DeploymentJob Double Job Name – One of the more peculiar conventions is the need to use the deployment job name twice in conditions with dependencies references. This is mentioned in the documentation as it states: Notice that in the condition of the test stage, build_job appears twice. I cannot tell how this conventionally makes sense. Without that single line in the documentation that has no explanation, I would have assumed deployment job references were utterly broken. The use of the dual Job Name is also needed in variable references to other deployment jobs as well using stageDependencies.
  • Secondary Variable Reference – Notice the last script output in the prodJob is attempting to demonstrate the build variable value. However, if using implicit dependencies and executing stages sequentially this value would be empty. If you need a variable created more than one stage prior, you have to explicitly use the “dependsOn” feature to add that stage as a dependency (along with the previous stage).

Definitely some odd conventions with variables between stages. Even still, I cannot wrap my head around entirely remembering them when I need them. This example pipeline is handy as a reference when you need to remember how to use and access variables in different stages based on different job types.

UPDATE: After some helpful support from the Microsoft team, the above information has been updated to indicate the need for the “dependsOn” property for secondary references, and the usage of dual job names specified in variable references.

Leave a Reply