Azure DevOps Pipelines: Leveraging OWASP ZAP in the Release Pipeline
Developer Support
In this blog App Dev Manager Francis Lacroix shows how to integrate OWASP ZAP within a Release pipeline, leveraging Azure Container Instances, and publish these results to Azure DevOps Test Runs.
As part of an organization’s automated Release pipeline, it is important to include security scans and report on the results of these scans. One tool used in the industry is the OWASP Zed Attack Proxy (ZAP). In this blog, we will integrate OWASP ZAP within a Release pipeline, leveraging Azure Container Instances, and publish these results to Azure DevOps Test Runs.
Assumptions
As this work is based on a PoC for a Premier Developer customer, this solution presented operates within certain assumptions.
- Leverage ACI to host OWASP ZAP on demand. The customer did not want to maintain an IaaS based installed of OWASP ZAP, nor did they have an AKS cluster to deploy the OWASP ZAP container into. They wanted an on-demand deployment to minimize management overhead of the security scanning tool.
- Import the scan results into Azure DevOps Test Runs. Since the customer already leverages Azure DevOps for automated test runs, they wanted the results of the OWASP ZAP scan in the same tool to present a single view of all test results.
- Run on a Microsoft Hosted Windows agent. The customer did not want to manage their own self-hosted agent(s), and requested this be done on a Windows based (VS 2017 at the time) agent.
Issues and Limitations
Based on the above assumptions, there were a few issues and limitations to overcome:
- Due to how ACI handles NATing, OWASP ZAP wasn’t able to bind to the container’s public address (know OWASP issue: see Behind NAT). As such, we were not able to leverage the API, which made us unable to use the task available in the Marketplace. An IaaS base solution could simply use the Talk in the Marketplace.
- OWASP ZAP’s report format is not natively supported by the PublishTestResults task. As such, we needed to convert it to a compatible format. A few options are available, we chose to use an XSL Template to convert it to a Nunit3 formatted results file.
- The work presented here is part of a Release Pipeline based on the customer needs. However, if it is to be reused in multiple pipelines, it would make more sense to set it up as a Task Group.
Building the Release Pipeline
The Release Pipeline itself is fairly simple. In our example, we will have one Artifact, which is an Azure Git artifact containing only the XSLTemplate used to transform the results file for publishing.
We have several Variables set on the Pipeline:
- ACI_RESOURCE_GROUP: The name of the Group the ACI instance will be deployed to.
- ACI_LOCATION: The geographical location to deploy to
- ACI_INSTANCE_NAME: The name of the actual ACI instance deployed in Azure
- ACI_STORAGE_ACCOUNT_NAME: The name of the Storage Account to hold the file share used to download the scan results report (more details below).
- ACI_SHARE_NAME: The name of the share where the report will be stored
- TARGET_SCAN_ADDRESS: The URL for OWASP ZAP to scan
Defining the Release Pipeline
Once the application portion of the Release pipeline has been configured, the security scan portion can be defined. In our example, this consists of 8 tasks, primarily using the Azure CLI task to create and use the ACI instance (and supporting structures).
Otherwise specified, all the Azure CLI tasks are Inline tasks, using the default configuration options.
Create Resource Group (if not created)
This task simply creates (if it doesn’t already exist) the Resource Group which all of the other services will be created in. It leverages the variables defined above and has a simple inline script.
rem Create the resource group az group create -l %ACI_LOCATION% -n %ACI_RESOURCE_GROUP%
Create Storage Account (if not created)
Similar to the previous task, this task simply create the Storage Account and File Share to be used with the OWAP ZAP Container Instance. This File Share will be mounted in the container instance and used to save the test results file generated by the security scan. The file will then be downloaded to be transformed and published to Azure DevOps Test Runs, as well as kept in archive for audit purposes.
rem Create the storage account with the parameters call az storage account create -g %ACI_RESOURCE_GROUP% -n %ACI_STORAGE_ACCOUNT_NAME% -l %ACI_LOCATION% --sku Standard_LRS rem Create the file share call az storage share create -n %ACI_SHARE_NAME% --account-name %ACI_STORAGE_ACCOUNT_NAME%
Create OWASP Container
This task does two things:
- Gets the storage key used to mount the File Share to the container
- Creates the actual OWASP ZAP container instance, based on the “zap2docker-stable” image in the Docker repository. Note: The container is created with a public IP address, but it will not be used in this example. This is for reference purposes only.
When creating the container instance, note the “–azure-file-volume-account-name”, “–azure-file-volume-account-key”, “–azure-file-volume-share-name” and “–azure-file-volume-mount-path”. These are used to mount the File Share specified in the variables (and created in the previous task) as “/zap/wrk” in the container instance. This is the location the scan reports a written to in the image.
rem Get the storage key call az storage account keys list -g %ACI_RESOURCE_GROUP% --account-name %ACI_STORAGE_ACCOUNT_NAME% --query "[0].value" --output tsv > temp.txt set /p STORAGE_KEY=<temp.txt rem Create the container call az container create -g %ACI_RESOURCE_GROUP% -n %ACI_INSTANCE_NAME% --image owasp/zap2docker-stable --ip-address public --ports 8080 --azure-file-volume-account-name %ACI_STORAGE_ACCOUNT_NAME% --azure-file-volume-account-key %STORAGE_KEY% --azure-file-volume-share-name %ACI_SHARE_NAME% --azure-file-volume-mount-path /zap/wrk/ --command-line "zap.sh -daemon -host 0.0.0.0 -port 8080 -config api.key=abcd -config api.addrs.addr.name=.* -config api.addrs.addr.regex=true"
Call the Baseline Scan
Once the container is created, the baseline scan will be called. OWASP ZAP offers a Baseline Scan as part of their Docker image. The ZAP CLI would also be an option if the Baseline is not sufficient.
The -x parameter will generate the XML report in the location mapped to the File Share above. We use the default config settings, but custom configurations could be provided through the file share.
rem Execute the baseline scan set "ZAP_COMMAND="/zap/zap-baseline.py -t %TARGET_SCAN_ADDRESS% -x OWASP-ZAP-Report.xml"" az container exec -g %ACI_RESOURCE_GROUP% -n %ACI_INSTANCE_NAME% --exec-command %ZAP_COMMAND%
Download the file
This task will download the “OWASP-ZAP-Report.xml” report to the local agent for conversion and publishing.
rem Get the storage key call az storage account keys list -g %ACI_RESOURCE_GROUP% --account-name %ACI_STORAGE_ACCOUNT_NAME% --query "[0].value" --output tsv > temp.txt set /p STORAGE_KEY=<temp.txt rem Download the file call az storage file download --account-name %ACI_STORAGE_ACCOUNT_NAME% --account-key %STORAGE_KEY% -s %ACI_SHARE_NAME% -p OWASP-ZAP-Report.xml --dest %SYSTEM_DEFAULTWORKINGDIRECTORY%\OWASP-ZAP-Report.xml
Convert Report Format
Instead of an Azure CLI task, this will be a PowerShell task to convert the OWASP ZAP report from its native format to a format (in this case, I chose NUnit3 since it was the closest) which can be uploaded to Azure DevOps Test Runs.
The script itself is fairly simple, but it relies on the template which can be found here: https://dev.azure.com/francislacroix/_git/CodeShare?path=%2FOWASPBlog%2FOWASPToNUnit3.xslt
The script itself is straightforward, set as Inline while leaving the rest of the parameters to their default value:
$XslPath = "$($Env:SYSTEM_DEFAULTWORKINGDIRECTORY)\_XSLTemplateFile\OWASPToNUnit3.xslt" $XmlInputPath = "$($Env:SYSTEM_DEFAULTWORKINGDIRECTORY)\OWASP-ZAP-Report.xml" $XmlOutputPath = "$($Env:SYSTEM_DEFAULTWORKINGDIRECTORY)\Converted-OWASP-ZAP-Report.xml" $XslTransform = New-Object System.Xml.Xsl.XslCompiledTransform $XslTransform.Load($XslPath) $XslTransform.Transform($XmlInputPath, $XmlOutputPath)
Publish Test Results
Now that the results have been converted, we can publish them using the built-in “Publish Test Results” task. After adding it, set the following parameters:
- Test Result Format: NUnit
- Test Result files: Converted-OWASP-ZAP-Report.xml
- Search Folder: $(System.DefaultWorkingDirectory)
- Test run title: OWASP Tests
Adjust the “Test Result files” if there are additional tests in the Release, or if the name was edited in the conversion script.
Destroy OWASP Container
Once all the scans are completed, the Container Instance can be destroyed. This is again an inline script using default settings.
az container delete -g %ACI_RESOURCE_GROUP% -n %ACI_INSTANCE_NAME% --yes
Closing Comments
This is a fairly simple use case for integrating OWASP ZAP in a Release pipeline, but the same concepts can be used for many other tools. I hope this helps you improve your automation and security of your software.
Light
Dark
14 comments
Really nice blog post but I thought https://docs.microsoft.com/en-us/azure/container-instances/container-instances-exec#restrictions would cause the az container exec step to fail, as its using arguments when calling zap-baseline.py ? I did attempt to create the above and it’s failing at that step due to this issue it appears. I’m using the latest version of azure cli.
You are correct. This should never have worked. I’ve posted a comment with a workaround and will submit an update to the post in the coming weeks.
I can confirm the problem as stated by the comment of Gereth Morris (Link). The main think that I’m trying to understand is exactly why adding parameters to the az container exec causes it to fail. I would expect that the executable line given would only execute in it’s entirety, not in parts. But that doesn’t seem to be the case.
I’m hostnestly not sure why adding the parameters is causing the failure, rather than ignoring them. It’s possibly a parsing issue. I didn’t spend too much time down that line of investigation, since I had to provide a solution to the customer. I’ve posted a comment with the workaround and will update the blog post in the coming weeks.
Same result here as Gareth and Jan-Rintje. Running the exec command generates a error that the file is not found, it is not handling the arguments”
Curious how it may have worked for your demo above, can you explain how this would work given the response below.
Any workaround to this ?
I wish I could tell you why it worked during my demos and part of the customer’s PoC. We only found out the issues when they tried to implement it in their production pipelines. I would guess this was due to some changes in the APIs the CLI calls that enabled it temporarily, but that’s only a guess on my part. As for the workaround, I’ve posted a comment and will update the actual post in the coming weeks.
Good morning. You are all quite correct, this should never have worked, as per https://docs.microsoft.com/en-us/azure/container-instances/container-instances-exec#restrictions. And yet it did, while it was in PoC with the customer. I can only assume that there were some changes in the APIs that temporarily allowed it to work.
There is a workaround, and it ended up being simpler in a way. Rather than calling the baseline scan from “az container exec”, we moved the call to “az container create”. The call to the “az container create” should now look like (note the added lines):
rem Create the containerset “ZAP_COMMAND=”/zap/zap-baseline.py -t %TARGET_SCAN_ADDRESS% -x OWASP-ZAP-Report.xml””
call az container create -g %ACI_RESOURCE_GROUP% -n %ACI_INSTANCE_NAME% –image owasp/zap2docker-stable –azure-file-volume-account-name %ACI_STORAGE_ACCOUNT_NAME% –azure-file-volume-account-key %STORAGE_KEY% –azure-file-volume-share-name %ACI_SHARE_NAME% –azure-file-volume-mount-path /zap/wrk/ –command-line %ZAP_COMMAND%
sleep 30
Three quick notes:
1) This works for this use case since the scan is only executed once, and the ACI is detroyed after the scan. If you have a use case where you want to run multiple scans, this may not be the approach for you. You can work around this by restarting the container, which will re-excute the command-line, but it’s not the cleanest solutio.
2) Note the “sleep 30” call. The reason is simple: The “az container create” call returns once the ACI is created, not once the commandline has been executed, so we need to delay the call to download the result file. You may need to adjust if you’re doing a different scan than the baseline.
3) If you have a lingering ACI from a preview failed run, I would recommend deleting it first. I had coworkers using this solution who applied the workaround without deleting the previous ACI, and they ran into odd behaviour. This was resolved with a restart of the ACI, but re-creating it will also prevent it.
Once I return to the office, I’ll submit an update to the article with these changes.
I attempted to format the command-line parameter with both the original startup value “zap.sh -daemon -host 0.0.0.0 -port 8080 -config api.key=abcd -config api.addrs.addr.name=.* -config api.addrs.addr.regex=true” and the additional ZAP script command to make the following
“zap.sh -daemon -host 0.0.0.0 -port 8080 -config api.key=abcd -config api.addrs.addr.name=.* -config api.addrs.addr.regex=true;/zap/zap-baseline.py -t https://google.ca -x OWASP-ZAP-Report.xml”
The release pipeline gives a false positive, saying the task is complete as you mentioned above due to it returning the call once the ACI is created. I gave it plenty of time, in one case I simply did not clean up the ACI after the run. Even after a 10 min wait I was still not seeing the xml output on the storage account.
Yet running the cmd manually through the terminal window via the azure portal does generate the xml without issue.
Full cmd for refernececall az container create -g %resourceGroupName% -n %ACI_INSTANCE_NAME% –image owasp/zap2docker-stable –ip-address public –ports 8080 –azure-file-volume-account-name %CALogStorageName% –azure-file-volume-account-key %storagekeyswasp% –azure-file-volume-share-name %owaspZapSharename% –azure-file-volume-mount-path /zap/wrk/ –command-line “zap.sh -daemon -host 0.0.0.0 -port 8080 -config api.key=abcd -config api.addrs.addr.name=.* -config api.addrs.addr.regex=true;/zap/zap-baseline.py -t https://google.ca -x OWASP-ZAP-Report.xml”
Nice blog. I have one question regarding converting the output file for publishing test result. How did you add OWASPToNUnit3.xslt to system default directory?
This is brilliant. I’ve ported it into an ARM template based on your very comprehensive blog post (thanks). It’s not currently possible to provision File Shares natively in ARM so I switched to using a Blob Store and container, which I could then use via a SAS token to publish the results. You can find it as an Azure Quickstart Template here:
https://github.com/nathankitchen/owasp-azure-arm
It’s not particularly widely tested though, if anyone is interested in taking it for a spin and providing feedback (pull requests, or find me on Twitter @nathankitchen) it’d be good to improve it for the wider community.
You had no problem calling the AZ Container EXEC Command? az container exec -g %ACI_RESOURCE_GROUP% -n %ACI_INSTANCE_NAME% –exec-command %ZAP_COMMAND%
There is a noted problem that you cannot pass multiple arguments through this “az container EXEC” command, written in MS Documentation. Can you explain how you were able to work around this problem?
I’m not sure of the details of the limitation, but I’m not calling az container exec, at least not directly. I’m specifying the command in the ARM template and letting Azure deal with it. If there’s a CLI limitation specifically, then I imagine this approach works because it’s not using CLI.
It might also because of how I’m forming the command: /bin/bash -c $ZAPCOMMAND, which might also be letting me chain the arguments. See the template here, around line 100.
Like the other comments I am also getting a file not found error. Has the post been updated with a fix? Is there a fix/work arround?
Excellent blog. Thanks for the steps.
If I want to use ZAP-CLI instead of the Baseline scan, what would be the command? I tried with the following command:
set “ZAP_COMMAND=”/zap/zap-cli quick-scan –self-contained –start-options ‘-config api.disablekey=true’ %TARGET_SCAN_ADDRESS%””
However, I have an error message: /zap/zap-cli: no such file or directory\
Where is zap-cli located?
Besides, how to generate the report from the command line? “az container create” seems to accept only one command line argument. I tried to concatenate commands but it fails.