Hello!

I have a pipeline that builds and publishes a wheel to Azure Artifacts in Azure DevOps, but I also wanted to take the wheel and publish it to some Azure Databricks workspaces. I could have made this one pipeline, but as there could be potentially a large number of workspaces to publish to I thought it best to write the terrafrom that will publish to the DBFS in a separate repo and make the publishing to workspaces a separate pipeline. That way we can add and remove workspaces to be included independently of creating a new version of the library and having to update any jobs on workspaces that currently exist to use a later version of the library.

To trigger a pipeline upon the creation of another pipeline is easy enough: you specify the triggering piepline as a pipeline resource in the pipeline you want to be triggered. So in my case the terraform repo (I’ll call this the “downstream pipeline”) that publishes the library to the Azure Databricks Workspaces will contain this snippet:

trigger: none

resources:
  pipelines:
  - pipeline: wheel
    source: build-wheel
    project: BZZZT
    trigger:
      branches:
        include:
          - main

The trigger at the top is entirely optional; but I really do only ever want this pipeline to be triggered by the completion of another pipeline. In this case the name of the triggering pipeline (I’ll refer to as the “upstream pipeline”) is called “build-wheel”, and as it is in a separate Azure DevOps project from the repo that contains the downstream pipeline I had to include the project name. I also added a condition to only ever trigger on the main branch. Thepipeline is used to refer to the pipeline from within the yaml file of the downstream pipeline, as we shall see.

And so because the downstream pipeline will only ever be triggered on completion of the upstream pipeline, I wanted to share the build number between the two of them so that it would be easier to track which upstream pipeline triggered the downstreeam pipeline. This ymal is form the upstream pipeline:

    variables:
    - name: Version.Major
      value: 2
    - name: Version.Minor
      value: 1
    - name: Version.Revision
      value: $[counter(variables['Version.Minor'], 0)]
    steps:
    - task: PowerShell@2
      inputs:
        targetType: inline
        script: |
          $buildwheelbuildnumber = "$($(Version.Major)).$($(Version.Minor)).$($(Version.Revision))"
          Write-Host "##vso[task.setvariable variable=buildwheelbuildnumber;isOutput=true]$buildwheelbuildnumber" 
          Write-Host "##vso[task.setvariable variable=buildwheelbuildnumber]$buildwheelbuildnumber" 
          Write-Host "##vso[build.addbuildtag]$buildwheelbuildnumber"
          $buildwheelbuildnumber = $env:BUILD_DEFINITIONNAME +'_'+ $buildwheelbuildnumber + '_' + $env:BUILD_SOURCEBRANCHNAME 
          Write-Host "##vso[build.updatebuildnumber]$buildwheelbuildnumber"
      displayName: Update Build Number and Tag Build

Above is the yaml that sets and defines the build number in the upstream pipeline. But as you can see I use the Azure DevOps predefined variables of BUILD_DEFINITIONNAME and BUILD_SOURCEBRANCHNAME in order to set the full build name and use some variables to create a semantic version, which is also used to set the semantic version on the wheel being published. But in addition to setting the build number, I now also use the logging command build.addbuildtag to add a tag to the upstream pipeline, the value of which is the version number.

So now that I have the tag being published after the upstream pipeline is run, and that I have the upstream pipeline as a pipeline resource set for the downstream pipeline, I can now access the tags using the az pipelines cmd from the az pipeline CLI. First I need to log in to Azure DevOps though, and happily I can use the $(System.AccessToken) which means no secret management! Once I have the value I can then set an environment variable to that value and then in the final step I can use that to set up the build number. This yaml is from the downstream pipeline:

      - bash: |
          echo ${AZURE_DEVOPS_CLI_PAT} | az devops login
          az devops configure --defaults organization=$(System.TeamFoundationCollectionUri) project="$(System.TeamProject)" --use-git-aliases true
        env:
          AZURE_DEVOPS_CLI_PAT: $(System.AccessToken)
        displayName: 'Login az'

      - task: Bash@3
        name: versionchaining
        inputs:
          targetType: inline
          script: |
            buildwheelbuildnumber=$(az pipelines runs tag list --project 'BZZZT' --run-id $(resources.pipeline.wheel.runID) --query "[]" -o tsv)
            echo "##vso[task.setvariable variable=buildwheelbuildnumber]$buildwheelbuildnumber"
        displayName: 'Get Version Number from build-wheel pipeline'

      - task: Bash@3
        name: UpdateBuildNumber
        inputs:
          targetType: inline
          script: |
            buildname="$BUILD_DEFINITIONNAME""_$BUILDWHEELBUILDNUMBER""_$BUILD_SOURCEBRANCHNAME"
            echo "##vso[build.updatebuildnumber]$buildname"
        displayName: 'Update BuildNumber'

So now the same version is used by both the upstream and downstream pipelines as well as the library. The parity of using the version number across the disparate parts of the whole system makes it much easier to track what was released by which pipeline.