2023-07

Display Custom Build Number in Azure DevOps Environments

Hello! I’ve been trying to wrap my head around how to get the Build Number to display in the environments status information in Azure DevOps. Environments use the Build Number to show which pipeline was last deployed. And so there is a distinction between a build number, which can include info like app name and branch name, and the semver number, which can be included with the build number. However the semver can be used when publishing artefacts from a pipeline, which doesn’t need all the info included in the build number: PowerShellGallery has strict versioning rules, and I don’t know about Pypi, but the convention is to use semantic versioning.

2023-07-10

Managing ADF Trigger Runtime State From Source Control

Hello! In a previous post, I wrote about how the runtime state of ADF triggers is ignored even if it is set in the ARM template. This means we have a couple of options when manging runtime state acorss different deployments of the same ADF. The first one is to use the UI of the ADF to set the trigger state. Even as someone who is quite vociferous against using UI’s to manage configuration, I think this is far from the worst idea.

2023-07-06

ARM Template for ADF Trigger Runtime State is Always Ignored

Hello! So, continuing the saga of deploying the ARM templates of ADFs that are generated by the npm package made avilable by Microsoft, I came across something that I was not aware of, and honestly I really should have been. Although runtimestate can be included in an ARM Template to deploy a Trigger, the runtimestate on the deployed ADF will not be altered to match what is in the ARM template.

2023-07-05

2023-06

Check for Failed ADF Linked Template Deployments in Azure DevOps

Hello! This series on ADF deployment management seems to be never ending and always evolving. In a previous post I shared some code that could be added to an Azure DevOps pipelines that would check for failed deployment messages. This was necessary as typically we’d not get a useful error back, and in some environments we did not have access to the Resource Group to check for deployment errors. Now that we have moved to making deployments using linked templates for ADF this code is no longer helpful, as it returns the same top error: each linked template has its own entry in the Deployment registry.

2023-06-23

PR Raised To Update PrePostDeploymentScript.Ver2.ps1

Hello! I’ve spent the day parallelising the entire PrePostDeploymentScript.Ver2.ps1, as well as merely the deleting deployments. I’ve also fixed a casing issue where the parameters file was not being found on the ubuntu-latest Azure DevOps Agent. This is the link for the Pull Request, and this is the link for the copy of the file in my forked repo. I have no idea if the changes will be accepted or not, but hopefully it will as it’s a good script it just needs a few tweaks to it.

2023-06-20

Parallelising Delete Linked Templates

Hello! The saga continues with deleting deployments for linked templates… come to find out that deleting linked templates serially can be extremely slow. So I updated my script to parallelise the deletion of the templates. Fortunately ForEach-Object has a parallel feature and can throttle the number of threads. It’s quite neat that each deplyoyment has all the info in it so the script block is able to reference the object for all it needs.

2023-06-19

More Woes with ADF Deployments

Hello! In a previous post, I wrote about some of the issues with the PowerShell script used to delete deployments. This is packaged as part of the npm package provided by Microsoft. We recently hit an issue with the number of resources in the ARM template generated by this npm package. Helpfully the package also creates the ARM templates as linkedtemplates. This was all fine, however the delete deployments section of this ill-fated PowerSHell script does not account for multiple deployments, which is exactly what happens when you use linked templates.

2023-06-17

The ADF PrePostDeploymentScript Has Some Issues

Hello! As part of the npm package provided by Microsoft, there is a PowerShell script called PrePostDeployment script. This script very helpfully sorts out the ADF prior to the ARM tempalte being deployed. This script however is not without its problems. Firstly when it attempts to stop triggers it does so by getting a list of all those that are not stopped. This line used to not include the -and $_.

2023-06-08

Check for Failed ADF Deployments in Azure DevOps

Hello! We’ve been using the npm package to build the ADF source code. The npm package, which is officially from Microsoft, will generate an ARM tempalte, which can then be deployed. As we’re using Azure DevOps it makes sense to use the AzureResourceManagerTemplateDeployment@3 task to deploy the ARM Template. All very nice, but we have found that owing to the sheer number of objects being deployed we get a very unhelpful error message back when it fails:

2023-06-08

2021-10

The Many Ways to Authenticate to Azure from Python In One Simple Class

Hello! I’ve been working on a script to login to Azure to interract with some resources that have been deployed by terraform. I want this script to be able to run locally on my machine and as well as part of a yaml pipeline. Initially I thought this was going to be quite tricky, in that I’d need to determine when to run the ClientSecretCredential or when to run one of the other authentication classes, like the AzureCliCredential or even the UsernamePasswordCredential.

2021-10-07

2020-04

Export-DatabricksFolder - Now With Threading

Hello! Last month I posted about the challenges of duration of releases getting tedious and what you can do about that. In a very similar theme, running a process like a build can also get tedious, and so you have to go looking for making improvements to a process before it really does start impeding people developing at a reasonable pace. I’m not going to repeat what I mentioned about three weeks ago, so go and have a read if you like, because what motivated me to make this change was getting bored of waiting for something to complete.

2020-04-15

Get All Users of Nested AAD Groups

Hello! I’ve spent the morning retrieving all of the users from a bunch of AAD Groups. The trouble was that groups were also members of the groups, so it was necessary to get the member of those groups as well. Actually, this was no trouble at all; it was a case of looping through and writing the users to a csv and looping through the sub-groups to get the users and groups etc etc.

2020-04-08

Azure DevOps Sample Repo and Pipelines for Artifacts

Hello! Last week I published a post about publishing a PowerShell module to the artifacts section of Azure DevOps. It was basically an amendment to a script that someone else had written. But what I really want to do is write a bit more of an in-depth post on the pipelines I have used to publish a PowerShell module as an artifact. I have created a repo on GitHub so that people can follow from home if they really want to or take scripts and stick them in their own repos.

2020-04-06

Publish PowerShell Module to AzDo Artifacts

Hello! I’ve been writing a PowerShell module that I want other developers to be able to use, so I decided to use the artifacts part of Azure DevOps as I’ve set up a build to run some tests and run PsScriptAnalyzer prior to publishing the module at the end of the build as an artifact. I figured that using the Artifacts to store my artifacts was sensible instead of leaving them hanging around old builds.

2020-04-03

Az-Connect on a Windows Server

Hello! I recently had to create a VM in Azure that was in the same region as a Data Lake I was working on as the network delay in running some Az PowerShell cmdlets from my laptop here in the UK was causing some issues. Logging in should be simple enough - if ($null -eq (Get-AzContext)){ Write-Host "No Azure Context... logging in. Check for login window behind this one, especially if you are using VSCode!

2020-04-01

2020-01

What's Going On In Deploy-AzureResourceGroup? - Part Zero

Hello! When creating Azure Resource Group projects, a PowerShell file is added to the project. The intended use of this file is to be used to deploy the Resource Group both locally and in a pipeline. This file is called Deploy-AzureResourceGroup. But at over 200 lines long, there seems to be quite a lot going on. And what exactly is going on? In short, it’s a helpful file, but tries to be all things to all people, with very little documentation.

2020-01-31

2019-03

ADF Activity Fails But Pipeline Run Succeeds - WTF

Hello! Since August last year I’ve had to pick up a number of new technologies, one of these being Azure Data Factory (Version 2 to be clear). And there’s a clear correlation between how much I enjoy using a technology compared to how long I’ve used it. And when it comes to ADF I’d say that I’ve hit a point of not enjoying it at all. Aside from the fact that it is practically impossible to test, the developer experience is a web page/drag and drop affair, and that the source control integration is rubbish (oh wow a blob of json to compare manually, what fun), the thing that’s really got me all angsty today is something totally unexpected, yet undoubtedly something someone somewhere would call “a feature.

2019-03-20

2018-04

PowerShell Snippet 13 - Skip Logging Into Azure Everytime With This One Simple Trick

Hello! I have been working an awful lot with ARM Templates and publishing resources using PowerShell. As part ofhte process we’re using a Service Endpoint to publish resoruces to Azure. So the login and subscription context are set by the endpoint itself. However, when running the scripts locally, it is necessary to login first. so you have two choices - login before running the script, which isn’t always possible, or include Login-AzureRmAccount in every script, and then obviously forget to remove it prior to checking in and wondering why releases takes forever…

2018-04-27

How Not To Create Your Service Principals for VSTS/ARM Connections

Hello! I’ve recently started a new project, and we’re going to use VSTS for the build and release pipeline. This includes deploying infrastructure from ARM templates. This is going to provide its own set of challenges, in that the VSTS Team is new and there is no authentication between VSTS and the Subscription the objects are deploying to. So it really is a “green field” project. But this is a good thing, as we get to deploy as much as we can from source control and limit the interaction with the Azure Portal.

2018-04-16

2016-09

VSTS Hosted Build Agent Specs

I was interested to know just what the hardware specifications of the hosted build agent is. So I added some PowerShell to read out the info below:  2016-06-29T09:23:31.3935358Z systemname Name DeviceID NumberOfCores NumberOfLogicalProcessors Addresswidth 2016-06-29T09:23:31.3935358Z ---------- ---- -------- ------------- ------------------------- ------------ 2016-06-29T09:23:31.3935358Z TASKAGENT5-0010 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz CPU0 2 2 64 2016-06-29T09:23:31.4095356Z Total memory: 7167.55078125 What piqued my interest greater was that this is the exact same spec for a D2 v2 box that is available via Azure.

2016-09-05

2016-07

Azure KeyVault and Extensible Key Management

I’ve been reading up on Azure KeyVault, and the more I read about it the more I like it. There is a Technet blog that posts news and other pieces of information. There’s even a forum for feedback and discussions. Utilising Azure KeyVault in SQL Server means that for Extensible Key Managementyou can now replace the need for a certificate stored in master with a key stored in the Azure Key Vault.

2016-07-08