By Richie Lee

Hello! And Happy New Year!

I’m going to preface this post with the fact that I am not sure if this is the intended way to get different versions of the Azure Powershell module running on a self-hosted nuild agent. It just feels wrong and contradicts the best practice of having modules in the $Env:PSModulePath. I’m sure there’s a very good reason that the hosted build agents have been set up this way, I’m just not privy to it.

Background

Not long ago we (as in Sabin IO) released a PowerShell module called adls2.folder.access.tools that would manage the Access Control Lists (ACL) on folders in the Gen 2 Azure Data Lake. This is an entirely different topic in and of itself and one that I’m not going to go into here other than providing some context for the problem/solution. But there is some Microsoft documention on the subject.

Anyway, in an ideal world when setting the ACL of a folder you should do it when it is empty, as if you want to the permissions to propagate throught the child folder/files then they can inherit the permissions of hte parent when they are created. To set the ACLs recursively on folders/files that already exist could take a long time, depending on the amount of files/folders that need the ACL applied to them. And seeing as data lakes store large amounts of objects it usually would lead to taking a long while. Of course we do not live in an ideal world and so lots of recursive permission setting is required. Fortunately the Az.Storage module recently added a new function that will set permissions recursively much quicker than before.

Problem

The function is only available in a version of the Az module that is not installed in the build agents. And consequently the Azure PowerShell task does not have that function installed and ready to use. From the doc page -

Azure PowerShell task uses Azure/AzureRM/Az PowerShell Module to interact with Azure Subscription. This issue occurs when the PowerShell module is not available on the Hosted Agent. Hence, for a particular task version, Preferred Azure PowerShell version must be specified in the Azure PowerShell version options from the following available list of versions.

Because the Az.Storage module requires a later version of Az.Account, and that is already in use in the Azure PowerShell tasks session, I could not just install and import the latest version of Az.Storage: I would need to terminate the session and update. But in the context of the Azure PowerShell tasks that deals with authentication to Azure, this would not be possible. This is hte error I get when I attempted to do that.

##[error]This module requires Az.Accounts version 2.2.3. An earlier version of Az.Accounts is imported in the current PowerShell session. Please open a new session before importing this module. This error could indicate that multiple incompatible versions of the Azure PowerShell cmdlets are installed on your system. Please see https://aka.ms/azps-version-error for troubleshooting information.

Solution

The Azure PowerShell task does accommodate for using versions that are not installed on the hosted agents. There are two arguments for setting the Azure PowerShell version -

new_build_numbers

So if I were to create a self-hosted build agent and install 5.3.0 then I could set azurePowerShellVersion to ‘OtherVersion’ and then set preferredAzurePowerShellVersion to 5.3.0. I felt this was a decent temporary solution until 5.3.0 is available on the hosted agents and I can switch them over ( I don’t want to have to go back to managing build agents thank you!)

So I set up the build agent, installed PowerShell, installed Az 5.3.0 using PowerShellGet, set the arguments above, ran the build…. and it failed! This is the log -

2021-01-12T10:18:15.8969140Z ##[section]Starting: Azure PowerShell script: analyticsukprodadls
2021-01-12T10:18:15.8976682Z ==============================================================================
2021-01-12T10:18:15.8977009Z Task         : Azure PowerShell
2021-01-12T10:18:15.8977305Z Description  : Run a PowerShell script within an Azure environment
2021-01-12T10:18:15.8977569Z Version      : 4.179.0
2021-01-12T10:18:15.8977817Z Author       : Microsoft Corporation
2021-01-12T10:18:15.8978349Z Help         : https://aka.ms/azurepowershelltroubleshooting
2021-01-12T10:18:15.8978691Z ==============================================================================
2021-01-12T10:18:16.8970408Z ## Validating Inputs
2021-01-12T10:18:16.8986469Z ## Validating Inputs Complete
2021-01-12T10:18:16.8986989Z ## Initializing Az module
2021-01-12T10:18:16.8987220Z Generating script.
2021-01-12T10:18:16.8987893Z ## Az module initialization Complete
2021-01-12T10:18:16.8988365Z ## Beginning Script Execution
2021-01-12T10:18:16.9005669Z [command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command . '/home/pdp/myagent/_work/_temp/22d38377-f453-4c50-ad0a-1c4005f8443e.ps1'
2021-01-12T10:18:16.9006904Z Exception: /home/pdp/myagent/_work/_tasks/AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62/4.179.0/Utility.ps1:42
2021-01-12T10:18:16.9013485Z Line |
2021-01-12T10:18:16.9015958Z   42 |  …             throw ("Could not find the module path with given version …
2021-01-12T10:18:16.9018920Z      |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-01-12T10:18:16.9019739Z      | Could not find the module path with given version.
2021-01-12T10:18:16.9020505Z 
2021-01-12T10:18:16.9054544Z ##[error]PowerShell exited with code '1'.
2021-01-12T10:18:16.9517453Z ## Script Execution Complete

How odd, I thought to myself. The error message references line 42, which must be part of the task as my script is much smaller than that, and so I checked the task on GitHub for that error message and sure enough there is it.

Long story short, the Az module is not stored in a directory that is included in $Env:PSModulePath. Instead it is installed in a folder under /usr/share. And when the task initialises it runs a set of PowerShell functions to find the module it needs. As you can see it makes use of AzurePowerShellVersion, which is set in the arguments above, to set the folder path. This is then imported into the session and the connection to Azure is made.

function Get-SavedModulePathLinux {
    [CmdletBinding()]
    param([string] $azurePowerShellVersion)
    $savedModulePath =  $("/usr/share/az_" + $azurePowerShellVersion)
    Write-Verbose "The value of the module path is: $savedModulePath"
    return $savedModulePath
}

I’m not going to go into why /usr/share was chosen as the location, because I don’t know. There’s documentation about that location, and I’ll leave that here. But this means that we need to get the module installed in there.

Assuming you have a build agent set up, and you’re logged in as a user with sudo permissions, and you have PowerShell installed on the machine, then these ar the steps required to get the module installed -

sudo pwsh
Find-Module -Name Az -Repository 'PSGallery' | Save-Module -Path '/usr/share' -Force -Verbose
cd /usr/share
mkdir az_5.3.0
cp -RT Az/5.3.0 az_5.3.0
cd .. 
cp -R share/Az share/az_5.3.0
cp -R share/Az.* share/az_5.3.0

Now when I ran the step it found the Az module 5.3.0 and the function could now be used.

Conclusion

I’m going to raise an issue on the function, or maybe the documentation, as I’m not sure if what I’m doing is correct, but in absence of any documentation I’m sticking with this as the way to go. The fact that it is undocumented, except for this blog post, leads me to suggest that maybe not many users have had to do this, or have given up and written their own tasks/functions as a workaround. But the solution above works if you want to use the Azure Powershell tasks still, like I do.