Managing ADF Trigger Runtime State From Source Control

Hello!

In a previous post, I wrote about how the runtime state of ADF triggers is ignored even if it is set in the ARM template. This means we have a couple of options when manging runtime state acorss different deployments of the same ADF.

The first one is to use the UI of the ADF to set the trigger state. Even as someone who is quite vociferous against using UI’s to manage configuration, I think this is far from the worst idea. Because state will be environment-specific, using the UI for each environment makes some sense.

The second option is to store the runtime state in source control. I’d advocate ignoring the state stored in the ADF ARM template entirely and using csv’s to store the runtime state for each environment you are deploying the triggers to. Chances are you’d want the triggers stopped in all but 1 or 2 environments, but even so a csv per environment allows us to test the process.

I’d also recommend storing state in a separate repo from the ADF code, so that runtime state management can be altered much faster. There are risks of drift between the triggers that exist and their type, but with a bit of effort you can automate some checks to make sure that the csv is up to date with triggers stored in the other repo.

I’ve written a script that will take a csv with three headers; Name, RuntimeState, TriggerType, and will compare against the triggers published to an ADF, and start those that need to be stopped and vice versa. It outputs a CustomObject with loads of info, and stores all sorts to files. This is something I’ve started doing recently as it is much easier to debug what is happening on a Pipeline Agent by spitting out objects used to make changes.

I’d really like to build it out into a module to make it easier to test. However for today this will do!

[CmdletBinding()]
param (
    [Parameter()][String]$ResourceGroupName,
    [Parameter()][String]$DataFactoryName,
    [Parameter()][String]$DesiredTriggerRuntimeStateCsvFile,
    [Parameter()][String]$ExportCsvOutputLocation = '.\',
    [Parameter()][int]$ThrottleLimit = 4,
    [Parameter()][switch]$WhatIf    
)

$ContemporaneousTriggerRuntimeStateCsvFile = (Join-Path $ExportCsvOutputLocation ContemporaneousTriggerRuntimeStateCsvFile.csv)
New-Item $ContemporaneousTriggerRuntimeStateCsvFile -Force | Out-Null
$ContemporaneousTriggerRuntimeStateCsvFile = Resolve-Path $ContemporaneousTriggerRuntimeStateCsvFile
$ContemporaneousTriggersFromAdf = Get-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName 
$ContemporaneousTriggersFromAdf | Select-Object -Property ResourceGroupName, DataFactoryName, Name, RuntimeState, @{Name='TriggerType';Expression={$_.Properties.GetType().Name}} | Export-Csv -Path $ContemporaneousTriggerRuntimeStateCsvFile

$DesiredTriggerRuntimeState = Import-Csv $DesiredTriggerRuntimeStateCsvFile
$ContemporaneousTriggerRuntimeState = Import-Csv $ContemporaneousTriggerRuntimeStateCsvFile

$TriggersToAlter = Compare-Object $DesiredTriggerRuntimeState $ContemporaneousTriggerRuntimeState -Property Name, RuntimeState, TriggerType -PassThru | 
Where-Object { $_.SideIndicator -eq '<=' } |
ForEach-Object {
    [PSCustomObject]@{
        Name              = $_.Name
        ResourceGroupName = ($ContemporaneousTriggerRuntimeState | Where-Object Name -eq $_.Name).ResourceGroupName
        DataFactoryName   = ($ContemporaneousTriggerRuntimeState   | Where-Object Name -eq $_.Name).DataFactoryName
        RuntimeState      = ($DesiredTriggerRuntimeState   | Where-Object Name -eq $_.Name).RuntimeState
        TriggerType       = ($ContemporaneousTriggerRuntimeState  | Where-Object Name -eq $_.Name).TriggerType
    }
}

$TriggersToAlterCsvFile = (Join-Path $ExportCsvOutputLocation TriggersToAlter.csv)
$TriggersToAlterCsvFile = Resolve-Path $TriggersToAlterCsvFile                    
$TriggersToAlter | Export-Csv -Path $TriggersToAlterCsvFile

$TriggersToStart = $TriggersToAlter | Where-Object { $_.RuntimeState -eq 'Started' }
$TriggersToStop = $TriggersToAlter | Where-Object { $_.RuntimeState -eq 'Stopped' }

$CurrentOutputEncoding = $OutputEncoding
$OutputEncoding = [ System.Text.Encoding]::UTF8 
Write-Host "`u{23F1} $($triggersToAlter.Count) triggers to alter `u{23F1}"
Write-Host "`u{23F5} $($triggersToStart.Count) triggers to start `u{23F5}"
Write-Host "`u{23F9} $($triggersToStop.Count) triggers to stop  `u{23F9}"
$OutputEncoding = $CurrentOutputEncoding

$triggersToStart | ForEach-Object -Parallel {
    if ($_.TriggerType -eq 'BlobEventsTrigger') {
        $timeout = 0
        Write-Host "[*] Subscribing $($_.Name) to events..."
        $AddAzDataFactoryV2TriggerSubscriptionParams = @{
            ResourceGroupName = $_.ResourceGroupName
            DataFactoryName   = $_.DataFactoryName
            Name              = $_.Name
        }
        if ($using:WhatIf.IsPresent -eq $true) {
            $AddAzDataFactoryV2TriggerSubscriptionParams.Add('WhatIf', $true)
        }
        $status = Add-AzDataFactoryV2TriggerSubscription @AddAzDataFactoryV2TriggerSubscriptionParams
        if ($using:WhatIf.IsPresent -ne $true) {
            while (($status.Status -ne 'Enabled') -or ($timeout -le 12)) {
                Start-Sleep -s 10
                $timeout ++
                $GetAzDataFactoryV2TriggerSubscriptionStatus = @{
                    ResourceGroupName = $_.ResourceGroupName
                    DataFactoryName   = $_.DataFactoryName
                    Name              = $_.Name
                }
                $status = Get-AzDataFactoryV2TriggerSubscriptionStatus @GetAzDataFactoryV2TriggerSubscriptionStatus
            }
            if ($status.Status -ne 'Enabled'){
                Write-Host "##[warning] Subscribing trigger $($_.Name) to events was not completed. Starting trigger $($_.Name) may fail."
            }
        }
    }
    Write-Host "[*] Starting trigger $($_.Name)..."
    $StartAzDataFactoryV2TriggerParams = @{
        ResourceGroupName = $_.ResourceGroupName
        DataFactoryName   = $_.DataFactoryName
        Name              = $_.Name
    }
    if ($using:WhatIf.IsPresent -eq $true) {
        $StartAzDataFactoryV2TriggerParams.Add('WhatIf', $true)
    }
    else  {
        $StartAzDataFactoryV2TriggerParams.Add('Force', $true)
    }
    Start-AzDataFactoryV2Trigger @StartAzDataFactoryV2TriggerParams
} -ThrottleLimit $ThrottleLimit

$triggersToStop | ForEach-Object -Parallel {
    if ($_.TriggerType -eq 'BlobEventsTrigger') {
        $timeout = 0
        Write-Host "[*] Unsubscribing $($_.Name) from events..."
        $RemoveAzDataFactoryV2TriggerSubscriptionParams = @{
            ResourceGroupName = $_.ResourceGroupName
            DataFactoryName   = $_.DataFactoryName
            Name              = $_.Name
        }
        if ($using:WhatIf.IsPresent -eq $true) {
            $RemoveAzDataFactoryV2TriggerSubscriptionParams.Add('WhatIf', $true)
        }
        $status = Remove-AzDataFactoryV2TriggerSubscription @RemoveAzDataFactoryV2TriggerSubscriptionParams
        if ($using:WhatIf.IsPresent -ne $true) {
            while (($status.Status -ne 'Disabled') -or ($timeout -le 12)) {
                Start-Sleep -s 10
                $timeout ++
                $GetAzDataFactoryV2TriggerSubscriptionStatus = @{
                    ResourceGroupName = $_.ResourceGroupName
                    DataFactoryName   = $_.DataFactoryName
                    Name              = $_.Name
                }
                $status = Get-AzDataFactoryV2TriggerSubscriptionStatus @GetAzDataFactoryV2TriggerSubscriptionStatus
            }
            if ($status.Status -ne 'Disabled'){
                Write-Host "##[warning] Unsubscribing trigger $($_.Name) to events was not completed. Stopping trigger $($_.Name) may fail."
            }
        }
    }
    Write-Host "[*] Stopping trigger $($_.Name)..."
    $StopAzDataFactoryV2TriggerParams = @{
        ResourceGroupName = $_.ResourceGroupName
        DataFactoryName   = $_.DataFactoryName
        Name              = $_.Name
    }
    if ($using:WhatIf.IsPresent -eq $true) {
        $StopAzDataFactoryV2TriggerParams.Add('WhatIf', $true)
    }
    else {
        $StopAzDataFactoryV2TriggerParams.Add('Force', $true)
    }
    Stop-AzDataFactoryV2Trigger @StopAzDataFactoryV2TriggerParams
} -ThrottleLimit $ThrottleLimit

$SwitchTriggers = [PSCustomObject]@{
    ResourceGroupName                         = $ResourceGroupName
    DataFactoryName                           = $DataFactoryName
    TriggersToAlterCsvFile                    = $TriggersToAlterCsvFile
    ContemporaneousTriggerRuntimeStateCsvFile = $ContemporaneousTriggerRuntimeStateCsvFile
    DesiredTriggerRuntimeStateCsvFile         = $ContemporaneousTriggerRuntimeStateCsvFile
    TriggersAltered                           = $TriggersToAlter
    TriggersStarted                           = $TriggersToStart
    TriggersStopped                           = $TriggersToStop
    WhatIfIsPresent                           = $WhatIf.IsPresent
}
Return $SwitchTriggers

bzzzt!

What's Going On?

Managing ADF Trigger Runtime State From Source Control