{"id":62352,"date":"2021-09-17T08:00:45","date_gmt":"2021-09-17T16:00:45","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/devops\/?p=62352"},"modified":"2021-09-14T11:27:46","modified_gmt":"2021-09-14T19:27:46","slug":"introducing-azure-devops-audit-stream","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/introducing-azure-devops-audit-stream\/","title":{"rendered":"Introducing Azure DevOps Audit Stream"},"content":{"rendered":"

Auditing is important in any environment and solution, to get a view on who is doing what, typically from a compliance or governance perspective. In most scenarios, the solution allows you to store auditing to a logfile. The downside is that nobody is really watching over the logs, until something goes wrong.<\/p>\n

Auditing is enabled by default for all Azure DevOps organizations, and cannot be turned off. This guarantees that you never miss an actionable event. Events get stored for 90 days after which they will get deleted automatically. One possibility though is, you can back up audit events to an external location to keep the data for longer than the 90-day period.<\/p>\n

The scenario I talk about in this article, is optimizing Audit Logging within Azure DevOps, by enabling the new, currently in public preview, ADO Audit Stream<\/strong>.<\/p>\n

What\/Why ADO Audit Stream<\/h2>\n

Instead of using the built-in Audit Logging, which allows DevOps Project Admins to retrieve Audit logging on screen, validate the activities happening in the Azure DevOps organization and optionally exporting to a JSON- or CSV-file type logfile, ADO Audit Stream allows for more transactional<\/strong> streaming of auditing information.<\/p>\n

By deploying an audit stream, auditing data can get sent to other services or solutions outside of Azure DevOps for further processing or inspection. Sending auditing data to other Security Incident and Event Management (SIEM) tools opens possibilities, such as alerting on specific auditing events, creating views on auditing data, and performing anomaly detection. It also allows you to store more than the default (and fixed) 90-days retention of auditing data.<\/p>\n

Audit streams represent a continuous stream of auditing logging events from your Azure DevOps organization to a stream target. Every half hour or less, new audit events are bundled and streamed to your targets. Remember that Audit Stream is an Organization<\/strong> setting, capturing all audit events from all projects within Azure DevOps.<\/p>\n

For now – remember it is in Public Preview<\/strong> so features might still change<\/em> – Audit Stream supports 3 different scenarios as Stream Targets:<\/p>\n

a) Azure Event Grid<\/a> b) Azure Monitor Logs<\/a> c) Splunk<\/a><\/p>\n

Scenario<\/h2>\n

In this article, I focus on Azure Event Grid<\/strong> integration only, to give you some idea about the concept of Auditing Stream<\/p>\n

The flow is the following:<\/p>\n

    \n
  1. An Auditing event is generated in Azure DevOps<\/li>\n
  2. Auditing Event Stream is configured for Event Grid<\/li>\n
  3. Event Grid transferts the message to an Azure Storage Account Queue (Event Grid can do a lot more though… like triggering Functions, WebHooks)<\/li>\n
  4. From Azure Storage Account Queue, the message could be send along to “any 3rd party monitoring tool”<\/li>\n<\/ol>\n

    \"Auditing_Flow_EventGrid\"<\/p>\n

    Walkthrough Setup<\/h2>\n

    Now you do have an idea what Auditing Stream is about, let me guide you through the main steps to see it in action.<\/p>\n

    Enable Streams<\/h2>\n
      \n
    1. To enable Streams, open the Organization Settings<\/strong> and navigate to the Auditing<\/strong> menu option. Besides Logs, you should also see Streams<\/strong><\/li>\n<\/ol>\n

      (If you don’t see Streams, make sure you have Azure DevOps Preview Features enabled, which can be validated as follows: From the Azure DevOps Portal, navigate to User Settings<\/strong> (upper right corner next to your user account bubble) and select Preview Features<\/strong>)<\/p>\n

      \"PreviewFeatures\"<\/p>\n

        \n
      1. From the Streams window, click the New Stream<\/strong> button, and notice the 3 different options of streaming integration available for now<\/li>\n<\/ol>\n

        \"StreamOptions\"<\/p>\n

          \n
        1. \n

          Select Azure Event Grid<\/strong><\/p>\n<\/li>\n

        2. \n

          You are asked for the Azure Event Grid Topic URL and Access Key.<\/p>\n<\/li>\n<\/ol>\n

          Deploy Azure Event Grid Topic<\/h2>\n

          Assuming you don’t have an Azure Event Grid Topic yet, use the following steps to create this resource:<\/p>\n

            \n
          1. From the Azure Portal (https:\/\/portal.azure.com), search for Event Grid Topic<\/strong> in the “Search Resources, services and docs”<\/em> field and click the + Create<\/strong> button to deploy a new Event Grid Topic. <\/li>\n<\/ol>\n
              \n
            • provide the necessary details around Subscription, Resource Group, unique Resource Name and Region for this resource to get created. <\/li>\n<\/ul>\n

              \"StreamOptions\"<\/p>\n

                \n
              1. Wait for the resource to get deployed successfully, and open it in the Azure Portal. Notice the Topic EndPoint<\/strong> under the Overview Blade of Event Grid Topic. This is the URL you need to provide in the Azure DevOps Stream settings for the Event Grid Topic. <\/li>\n<\/ol>\n

                \"StreamOptions\"<\/p>\n

                  \n
                1. Navigate to the Access Keys<\/strong> option and copy the Key 1<\/strong> content to the corresponding field in the Azure DevOps Stream settings for Event Grid Topic.<\/li>\n<\/ol>\n

                  \"StreamOptions\"<\/p>\n

                    \n
                  1. The Event Grid settings for Azure DevOps Stream look similar to my sample setup:<\/li>\n<\/ol>\n

                    \"StreamOptions\"<\/p>\n

                      \n
                    1. Save the settings, which creates the Stream integration.<\/li>\n<\/ol>\n

                      \"StreamOptions\"<\/p>\n

                      If you would now trigger an Azure DevOps Event, it will get streamed to Azure Event Grid. However, Event Grid has no subscriptions yet, so it has no idea what to do with this receiving information.<\/p>\n

                      Event Grid supports different EndPoints<\/strong> such as Azure Functions, WebHooks, Service Bus, Storage Queue as targets.<\/p>\n

                      \"StreamOptions\"<\/p>\n

                      Create Storage Account Queue with Managed Identity<\/h2>\n

                      In this example, I will use a Storage Account Queue, but if you have experience with any of the other options, feel free to give that a try.<\/p>\n

                      In order to define this as an EndPoint, let’s create the Storage Account Queue.<\/p>\n

                        \n
                      1. \n

                        Create a new Azure Storage Account v2, in the same Location and Resource Group as where the Event Grid Resource got created earlier.<\/p>\n<\/li>\n

                      2. \n

                        Once the Storage Account is created, navigate to Data Storage \/ Queue<\/strong> in the Storage Account blade, and create a new Queue. I called mine devopsmessages<\/strong> but it can have any name you want.<\/p>\n<\/li>\n<\/ol>\n

                        \"StreamOptions\"<\/p>\n

                        To allow the Event Grid Resource to communicate with the Azure Storage Account Queue, I’m using Azure Managed Identity<\/strong>. This relies on the Azure Identity Service Principal concept to authorize both services to communicate with each other.<\/p>\n

                          \n
                        1. Navigate back to the Event Grid Topic created earlier, and select Identity<\/strong> under the settings. Next, select System Assigned<\/strong>, which creates a 1-to-1 mapping between the Event Grid Resource and its Azure Identity Service Principal object. Save<\/strong> the changes.<\/li>\n<\/ol>\n

                          \"StreamOptions\"<\/p>\n

                            \n
                          1. Navigate back to the Azure Storage Account you created earlier, and select Access COntrol (IAM)<\/strong> from the settings. Specify the following Role Permissions:<\/li>\n<\/ol>\n
                              \n
                            • Role<\/strong>: Storage Queue Data Contributor<\/li>\n
                            • Assign access to<\/strong>: User, Group, Service Principal<\/li>\n
                            • Select<\/strong>: <\/the><\/li>\n<\/ul>\n

                              \"StreamOptions\"<\/p>\n

                              This provides the necessary Role Based Access Control, to allow Event Grid to communicate with the Storage Account Queue, and creating messages in the queue itself.<\/p>\n

                              Configure Event Grid Subscription<\/h2>\n

                              In this last step, let’s configure an Event Grid Subscription, which will “listen” for events on the Azure Audit side.<\/p>\n

                                \n
                              1. Open your Event Grid Topic blade, select Overview<\/strong> and click + Event Subscription<\/strong>. Complete the necessary fields with the correct parameters:<\/li>\n<\/ol>\n
                                  \n
                                • Name<\/strong>: a unique name for the Event Grid Subscription <\/li>\n
                                • Event Schema<\/strong>: Event Grid Schema<\/li>\n
                                • EndPoint Type<\/strong>: Storage Queues<\/li>\n
                                • EndPoint<\/strong>: The Storage Account Queue object<\/li>\n
                                • Managed Identity Type<\/strong>: System Assigned <\/li>\n<\/ul>\n

                                  Here is a screenshot of the settings I used for this article setup:<\/p>\n

                                  \"StreamOptions\"<\/p>\n

                                  Monitoring Audit Stream info coming in<\/h2>\n

                                  Event Grid and its subcomponent Event Grid Subscription, provide detailed metrics information directly from within the Resource Blade.<\/p>\n

                                    \n
                                  1. \n

                                    Generate<\/strong> a few Events on the Azure DevOps side, by for example checking the Audit Log, Trigger a Pipeline run or change Permissions on a Project.<\/p>\n<\/li>\n

                                  2. \n

                                    After only a few minutes, you should see the events showing up in the Event Grid Topic<\/strong> Overview section:<\/p>\n<\/li>\n<\/ol>\n

                                    \"StreamOptions\"<\/p>\n

                                      \n
                                    1. At the bottom of the chart, select the Storage Queue Subscription<\/strong> object, which redirects you to a more granular dashboard view for the subscription object.<\/li>\n<\/ol>\n

                                      \"StreamOptions\"<\/p>\n

                                        \n
                                      1. Last, select the Storage Queues EndPoint<\/strong> underneath the chart, which opens the Storage Account Queue, where you can see the actual DevOps Audit information in a queue message. <\/li>\n<\/ol>\n

                                        \"StreamOptions\"<\/p>\n

                                        This confirms the setup is complete and Azure DevOps Audit Stream to Event Grid and Storage Queue EndPoint is working as expected.<\/p>\n

                                        Summary<\/h2>\n

                                        In this article, I wanted to introduce you to a new Public Preview<\/strong> feature available in Azure DevOps, namely Audit Stream<\/strong>. This feature allows an organization to stream Auditing information out of Azure DevOps Logs to another solution, for example Azure Event Grid, Azure Monitor Logs or Splunk.<\/p>\n

                                        This article covered Event Grid with Storage Account Queue as an example scenario to get you started and get an understanding of the core capabilities. Since Auditing is a crucial component of a healthy platform, I would recommend you to keep your eyes out for any updates on this preview feature, to see it moving to General Availability. If you should have any questions in meantime, do not hesitate to reach out [petender] @ [microsoft.com] or [pdtit] on Twitter.<\/p>\n","protected":false},"excerpt":{"rendered":"

                                        Auditing is important in any environment and solution, to get a view on who is doing what, typically from a compliance or governance perspective. In most scenarios, the solution allows you to store auditing to a logfile. The downside is that nobody is really watching over the logs, until something goes wrong. Auditing is enabled […]<\/p>\n","protected":false},"author":45107,"featured_media":45953,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[224],"tags":[],"class_list":["post-62352","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure"],"acf":[],"blog_post_summary":"

                                        Auditing is important in any environment and solution, to get a view on who is doing what, typically from a compliance or governance perspective. In most scenarios, the solution allows you to store auditing to a logfile. The downside is that nobody is really watching over the logs, until something goes wrong. Auditing is enabled […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/62352","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/45107"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=62352"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/62352\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/45953"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=62352"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=62352"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=62352"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}