Well don’t I look silly. This blog should have been published some time ago. And I mean a time before that pesky little global pandemic came about. But, such is life, so here’s the belated content as a follow up to when I previously shared a run down of ARM Custom Script extension vs Desired State Configuration extension and why I would use one over the other. I hinted at this follow up blog and a solution to execute multiple Custom Script Extension (CSE) scripts post deployment. Unfortunately a myriad of things got in the way and I’m publishing this a year some time later… 🤦
To begin, there is a slight out of the box problem with Custom Script Extensions (CSE) for Windows and Custom Script for Linux. Azure Resource Manager has a bit of an annoying limitation whereby IaaS compute deployments limit the number of CSE’s that can be used. To my knowledge thats a single type: “CustomScriptExtension” per template file for Windows and a single type: “CustomScript” for Linux instances.
Only one version of an extension can be installed on a VM at a point in time. Specifying a custom script twice in the same Azure Resource Manager template for the same VM will fail.
I was running into this often with error of “code: Bad Request, message: Multiple VMExtensions per handler not supported for OS type ‘Windows’.”
Now, thats a bit of an annoying limitation. So on a previous project I needed to do some Azure DevOps-ing on some new Azure IaaS infrastructure for a customer. What I needed was to execute multiple custom script extensions’s and to do that for not only Windows but also for Linux and the best part was that I needed to do it from a single ARM template file.
There’s many ways to skin a cat, as they say. So this is one solution that I used for this specific use case. There’s great tools like Ansible and Packer that can build sexy images to be used from an Azure Shared Image Gallery. Perhaps thats a blog topic for another day. For now, here’s one method to the madness.
The main problem that I needed solved was that the pipeline and ARM template IaaS deployment needed to be re-used across multiple Releases and, again, it had to cater for Windows and Linux resources from the one ARM template (with IF statements and smarts based on parameter inputs).
So, depending on what Variable Groups in Azure DevOps and specific set of parameters were selected, a template Pipeline may deploy either Windows or Linux instances or a combination of the two. To keep things in line with existing and established patterns, as I mentioned earlier, my constraints were using a single and uniform ARM template was needed.
After digging through the Azure Quick Start Templates for possible solutions, there was something quite useful that I stumbled upon and grabbed, and worked into the IaaS ARM template. In this blog I’ll share that solution and some gotchas to avoid.
To deliver on requirements and align with the existing deployment pattern, I had the means to use Azure DevOps and leverage Pipline multiple Tasks. However, the deployment process would then mean that the initial VM deployment would take place, then post deployment those additional tasks would execute. Some blockers meant that this approach was not viable. (for now, see the end of the blog for and update on this)
Therefore, rather than relying on the pipeline to do this, I relied on the ARM deployment to do this for me. Initially I did this leveraging two ARM Templates:
I personally like to separate templates where possible from low touch to high touch points to minimise blast radius for changes. More on this a little later. So I was on the right path but I then separated out the deployment tasks a bit further. After tinkering with how to get the deployment working, I managed to get a breakthrough that resulted in the below outcome.
Just before I get to the solution, I did want to call out that there are designated script extensions (" Microsoft.Compute/virtualMachines/extensions “) that can be associated with VM deployments. Things like OMS Agent deployment or domain join to name a couple. These can also be multiple assignments to a VM with no problem - as well as having custom script extensions. Just repeating myself - it was the custom script extension that there are limitations.
In an ARM template, when deploying a compute instance, what is represented after the compute block for deploying a custom script extension is 'Microsoft.Resources/deployments'. It is here that we can look at adding multiple instances of these blocks to allow multiple custom script extensions to run. Below is an example of what the CSE post deploy block looks like for a single CSE:
"variable": {
"PostDeploy": "[concat(parameters('artifactsLocation'), '/ChildTemplate/PostDeploy-CustomScriptExtension-Template.json', parameters('artifactsSasToken'))]",
"PostDeploymentScript01": "[concat(parameters('artifactsLocation'), '/Scripts/PowerShellAwesomeScript.ps1', parameters('MySasToken'))]",
"PostDeploymentScript01Command": "[concat('powershell -ExecutionPolicy Bypass -file ', '.\\Scripts\\', 'PowerShellAwesomeScript.ps1', ' -exampleParameter ', parameters('exampleParameter'), ' -exampleParameter ', parameters('exampleParameter'))]",
"PostDeploymentScript02": "<example here>",
"PostDeploymentScript02Command": "<example here>",
},
"resources": [
{
"name": "[concat(variables('deploymentPrefix'), '-UniqueNameForThisCSEDeployment')]",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2018-05-01",
"dependsOn": [
"<example the VM deployment or a previous CSE that is a pre-req to this>",
],
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "[variables('PostDeploy')]",
"contentVersion": "1.0.0.0"
},
"parameters": {
"location": {
"value": "[parameters('regionToDeploy')]"
},
"extensionName": {
"value": "CSE"
},
"vmName": {
"value": "[variables('vmName')]"
},
"fileUris": {
"value": [
"[variables('PostDeploymentScript01')]"
]
},
"commandToExecute": {
"value": "[variables('PostDeploymentScript01Command')]"
},
"isWindowsOS": {
"value": "[variables('isWindowsOs')]"
}
}
}
},
{
"name": "[concat(variables('deploymentPrefix'), '-UniqueNameForThisCSEDeployment')]",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2018-05-01",
"dependsOn": [
"<example the VM deployment or a previous CSE that is a pre-req to this>",
],
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "[variables('PostDeploy')]",
"contentVersion": "1.0.0.0"
},
"parameters": {
"location": {
"value": "[parameters('regionToDeploy')]"
},
"extensionName": {
"value": "CSE"
},
"vmName": {
"value": "[variables('vmName')]"
},
"fileUris": {
"value": [
"[variables('PostDeploymentScript02')]"
]
},
"commandToExecute": {
"value": "[variables('PostDeploymentScript02Command')]"
},
"isWindowsOS": {
"value": "[variables('isWindowsOs')]"
}
}
}
},
]
In this example I’ve included two CSE deployments to show how you can have multiple, one after the other. In the syntax itself, t he resources feature a bunch of parameters that are important to call out and I wanted to leave these in there to get a succinct overview of whats possible. Ignoring the example named variable values, what is important is:
"Microsoft.Resources/deployments" block. The goodness come by way of being able to repeat these “/deployments” blocks multiple times to allow for multiple post deployment scripts to be executed for a given VM instance.As I said before, this particular deployment block can be copied and updated to run N times in a single ARM template, either Windows or Linux, which was quite sexy.
Just a word of warning around the number of deployments. It’s not infinate. I don’t anticipate anyone ever reaching this number, but just calling out for reference - there is a limit of 800 deployments per resource group. Moreover, there is a limit of 800 resources per deployment. Those are some big numbers and I would say that shouldn’t come into play around this VM and custom script extension solution. Just keep it in the back of your head so you’re not expecting too much of Azure.
That secret sauce that makes this all possible is the embedded/child template that I’ve so creatively named PostDeploy-CustomScriptExtension-Template.json in this example. Now, here’s what the embedded PostDeploy-CustomScriptExtension-Template.json looks like:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"location": {
"type": "String"
},
"extensionName": {
"type": "string",
},
"vmName": {
"type": "string"
},
"fileUris": {
"type": "array"
},
"commandToExecute": {
"type": "string"
},
"isWindowsOS": {
"type": "bool"
}
},
"variables": {
"extensionNameWindows": "[concat(parameters('extensionName'), '-windows')]",
"extensionNameLinux": "[concat(parameters('extensionName'), '-linux')]"
},
"resources": [
{
"condition": "[parameters('isWindowsOS')]",
"type": "Microsoft.Compute/virtualMachines/extensions",
"apiVersion": "2017-03-30",
"name": "[concat(parameters('vmName'), '/', variables('extensionNameWindows'))]",
"location": "[parameters('location')]",
"dependsOn": [
],
"properties": {
"publisher": "Microsoft.Compute",
"type": "CustomScriptExtension",
"typeHandlerVersion": "1.8",
"autoUpgradeMinorVersion": true,
"settings": {
"fileUris": "[parameters('fileUris')]",
"commandToExecute": "[parameters('commandToExecute')]"
},
"protectedSettings": {
}
}
},
{
"condition": "[not(parameters('isWindowsOS'))]",
"type": "Microsoft.Compute/virtualMachines/extensions",
"apiVersion": "2017-03-30",
"name": "[concat(parameters('vmName'), '/', variables('extensionNameLinux'))]",
"location": "[parameters('location')]",
"dependsOn": [
],
"properties": {
"publisher": "Microsoft.Azure.Extensions",
"type": "CustomScript",
"typeHandlerVersion": "2.0",
"autoUpgradeMinorVersion": true,
"settings": {
"fileUris": "[parameters('fileUris')]"
},
"protectedSettings": {
"commandToExecute": "[parameters('commandToExecute')]"
}
}
}
]
}
In this example embedded or child template we have a selection process to either deploy a Windows Custom Script Extension or a Linux Custom Script. The two are different because the process to execute them is slightly different given the two differnet operating systems. But, with either OS, the process to loop through various CSEs via the parent template and the “/deployments” block is the same.
Note: from a parent to child template perspective, the parent attributes that are assigned to a resource are carried through as paramaters to the child.
There is a restriction that makes this tricky to execute and something I found the hard, frustrating way. The deployment itself is not to tricky. Running in a copy loop isn’t that bad either.
The kicker and what was hard to work out was that the name of the deployment needed to be consistent. That is "extensionName": {"value": "CSE"}, needed to always be CSE if that was being copy looped. This is because when ARM executes this you’ll see that the deployment, say via the Azure Portal under the resource, will use this name as the main deployment for the custom script extentions. Each iteration or individual CSE will be shown under the main CSE deployment using the resource “name” for that deployment, for example, taking that from above would mean the name is: "name": "[concat(variables('deploymentPrefix'), '-UniqueNameForThisCSEDeployment')]",.
As I said before this deployment block can be repeated a number of times in an ARM template, with variables and parameters renamed with say number suffixes so they’re unique, which then means that in the Azure Portal under the ‘CSE’ deployment will be a label for each subscript that runs, for example here ExecutePowerShellBootstrap01 would be one.
After deploying this solution and usage for some time, it became tricky to manage a single ARM template as the overall size was causing administrative overhead. The next iteration of it split the main ARM template into two distint operations and aligned to two separate Azure DevOps Release stages - VM Deployment and Post Deployment.
VM Deployment - The first stage was to deploy the VM itself and any adjacent components - for example a NIC NSG. This allowed for the complexities of catering for either a Windows or Linux VM to be more streamlined to develop - as this template focussed on just building a VM (or several, as inputs could determine one or many VMs).
Post Deployment - From there, the post deployment activities were moved into its own stage in the pipeline and all the custom script extensions were moved there. Going back to my preference to split out files - this meant that from an administrate perspective it also made CSEs easier to setup. That is, there’s a lot of repetition when it comes to the ARM template blocks and overall syntax, but its the parameters and variables and that management that is more difficult to not overalp and to not have ambiguous names that anyone else wanting to use the template cant pick up.
I’ve not included any example here as this iteration was more of an Azure DevOps configuration than associated with custom script extensions and repeatability. However, that detail can be saved for another blog, one day.
I did want to call out that the overall single ARM template solution and meeting the customers requirements for that approach was executed very elegantly in the first place was created by a fantastic Azure consultant - Tye Barker.
I worked with Tye on this particular VM deployment project and contributed my part in getting the custom script extension process to execute numerous times to cater for the ~5 or so post deployment scripts that needed to run.
In the end, I believe the customer moved to a new approach for virtual machine deployment, but none the less, this was interesting and I wanted to share how to achieve that if it helps!?!
How can this be done in Azure Bicep? Expect a follow up that goes into detail on how to do that. Cheers!