Azure DevOps Self-Hosted VMSS Agents Part 3

Creating an Azure DevOps Virtual Machine Scale Self-Hosted Agent Pool.

Overview

In part1 and part2 of this blog series we created an Ubuntu 20.04 VM image using the same image that Microsoft use for Azure DevOps hosted agents and then went on to deploy an Azure Virtual Machine Scale Set pointing at that image stored in an Azure Compute Gallery. In this post we will create an Azure virtual machine scale set agents pool and run some pipelines using our Virtual Machine Scale Set.

Requirements

You will need the same Azure requirements as needed in the previous posts in this series plus an Azure DevOps Organization that you have privileged access to e.g. a member of the Project Collection Administrators. The reason for this is that we will be creating a Project, AzureRM Service Connection and various other elements within Azure DevOps to demonstrate in this post. You will need to create a new Personal Access Token (PAT) in Azure DevOps in a user context that has the aforementioned privileged access that we will use in the steps below. Normally for creating an agent pool into an existing project only Agent Pools - Read & manage is required but as we will automate various things within Azure DevOps we will need a PAT with Full access permissions.

The installed Azure CLI will need to have the Azure DevOps Extension for Azure CLI enabled prior to running the scripts i.e.:

1az extension add --name azure-devops

Note: this is already installed in the container image we will use in the steps below.

Create Azure DevOps Project

The below steps will create the following items described in the Part 1 TLDR i.e.:

• A new Azure DevOps project.
• A new Azure DevOps AzureRM service connection.
• A new Azure virtual machine scale set agents pool using the VMSS created above.
• New Azure DevOp Pipelines:
  1. Basic VMSS test pipeline.
  2. Example basic Terraform pipeline configured to use Azure Managed Identity.
  3. Scheduled pipeline to run the Ubuntu 20.04 Linux VM image process.

This will be done using the same container approach used in part 1.

First run our deployment environment:

1docker run -it --rm ghcr.io/tonyskidmore/azure-tools:latest /bin/bash

Within the container clone the repository that contains the scripts that will generate our Azure resources:

1git clone https://github.com/tonyskidmore/azdo-vmss-virtual-environments.git
2cd azdo-vmss-virtual-environments

Export the necessary environment variables that the scripts use to authenticate to Azure, substituting the values below with your Azure service principal details.

1 export ARM_SUBSCRIPTION_ID=00000000-0000-0000-0000-000000000000
2 export ARM_TENANT_ID=00000000-0000-0000-0000-000000000000
3 export ARM_CLIENT_ID=00000000-0000-0000-0000-000000000000
4 export ARM_CLIENT_SECRET=AAABjkwhs7862782626_BsGGjkskj_MaGv

Additionally you will need to export two further environment variables for Azure DevOps. Replacing the below with your Azure DevOps organization URL and the Personal Access Token mentioned in the Requirements section:

1 export ADO_ORG=https://dev.azure.com/tonyskidmore
2 export AZURE_DEVOPS_EXT_PAT=5oxbb6op2vzwppglbfr1d321i3lk21ysjsh4c42eby99yepowopd

Launch the script to deploy Azure DevOps resources:

1scripts/ado-setup.sh

The last output from the script should be Creating Azure DevOps pool: ve-vmss followed by a JSON dump of the elasticPool creation response. Everything should be good to go!

Created Azure DevOps Resources

By default, the following resources will be created in the target Azure DevOps organization as part of the scripted execution:

Note: In our deployment of VMSS there are no instances created by default. VMSS instances are only deployed when one of the pipelines configured to use the VMSS pool are run i.e. vmss-test or terraform-example.
After a period of inactivity any instances will be deleted.

Many of the values that control the Azure DevOps resource creation can be changed by exporting environment variables prior to running the script.

Default value for ADO_REPO_SOURCE: https://github.com/tonyskidmore/azdo-vmss-virtual-environments

Azure DevOps Review and Testing

Now that we have our scripts/ado-setup.sh has been executed we can explore in Azure DevOps what has been created, check the following:

• A new project has been created, by default ve-vmss.
• A new agent pool has been created in Organization Settings - Pipelines - Agent pools and Project Settings - Pipelines - Agent pools, by default ve-vmss. Open the ve-vmss agent pool and selecting the Settings tab you will see the settings that have been set by default by the script. You will also see at the bottom of the Settings page that the Virtual machine scale set: is configured to our VMSS, created in Part 2 of this series.
• Open the project and in Project Settings - Pipelines - Service connections confirm that a ve-vmss AzureRm service connection exists, be default ve-vmss. Click on the service connection and then select Edit, scroll down and click Verify to check that the verification is successful.
• Navigate to Repos and confirm that the repository contains the azdo-vmss-virtual-environments content.
• Navigate to Pipelines and check that you you can see the pipelines vmss-test, terraform-example and image-build.
• Run the vmss-test pipeline. After running the pipeline check in the Azure Portal for the Virtual Machine Scale Set vmss-azdo-agents-01 - Instances, eventually you will see that an instance is being created based on the fact that a pipeline configured to use the VMSS agent pool has been executed. Confirm that the vmss-test pipeline completes successfully. Open the vmss-test pipeline. From the vertical ellipses in the top right (next to Run pipeline) select Schedule runs. You will see that this is scheduled to run each hour, I used this in testing to ensure that pipelines would run successfully and VMSS instance span up on demand. You can leave this in place or amend the schedule in the YAML pipeline to avoid the cost of bringing up a VM instance every hour or just delete the pipeline after you have tested running it.
• Run the terraform-example pipeline. Note that this pipeline is setup to use the Azure AD Managed Identity assigned to the VMSS as the method of authentication. It does not use Terraform remote state but instead just uses pipeline cache to move state between stages (just to avoid having to configure remote state as we are not persisting anything), the pipeline will create and then destroy a test resource group (rg-ado-vmss-example) in the target subscription. You will be prompted to validate the Plan and Destroy stages (this manual validation will timeout and resume if you do not manually validate and resume). Confirm that the terraform-example pipeline completes successfully.
• Confirm that VMSS instances are scaled to zero after ~30 minutes of pipeline inactivity.
• Open the image-build pipeline. From the vertical ellipses in the top right (next to Run pipeline) select Schedule runs. Note that the pipeline is set to run daily overnight. You can change the schedule or even delete the pipeline if you don't want it to run on the defined schedule. You can also run the pipeline to check if the latest virtual-environments ubuntu release builds successfully. When a successful build completes the built image will be added as an auto-incrementing VM image version to the Azure Compute Gallery Virtual image definition. By default, the script is configured to only maintain 2 existing VM image versions (excluding the image that is currently assigned to the VMSS). Rerunning the scripts/vmss-create.sh will and setting AZ_ACG_VERSION environment variable will update the image version used by the VMSS.

Conclusion

Hopefully you have been able to follow this blog series through successfully and have been able to successfully create an autoscaling Azure DevOps VMSS based Ubuntu Linux agent pool. You should be able to take the basic concepts introduced here and tailor them for your own purposes. Maybe you want to tweak the Azure DevOps VMSS agent settings to have a larger maximum number of instances, schedule updates so you have X number of agents during the daytime and scale to zero in the evenings and weekends or maybe use Terraform instead of the Azure CLI to perform the Infrastructure as Code side of things?

Please remember to delete the Azure resources and Azure DevOps project when you are finished to avoid any unnecessary cost.

It would be appreciated if you would let me know if this blog post series was of use to you or you hit any issues. Please contact me using any of the following methods: comments section on a post, Twitter or raise an Issue.

Thanks!