{"id":2700,"date":"2023-05-30T08:20:14","date_gmt":"2023-05-30T15:20:14","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/azure-sdk\/?p=2700"},"modified":"2023-05-30T08:20:14","modified_gmt":"2023-05-30T15:20:14","slug":"resource-centric-log-queries","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/azure-sdk\/resource-centric-log-queries\/","title":{"rendered":"Resource-centric log queries with the Azure Monitor Query libraries"},"content":{"rendered":"<p>In October 2021, the Azure SDK team shipped the <a href=\"https:\/\/devblogs.microsoft.com\/azure-sdk\/announcing-the-new-azure-monitor-query-client-libraries\/\">initial stable release<\/a> of the Azure Monitor Query client libraries for <a href=\"https:\/\/learn.microsoft.com\/dotnet\/api\/overview\/azure\/Monitor.Query-readme\">.NET<\/a>, <a href=\"https:\/\/learn.microsoft.com\/java\/api\/overview\/azure\/monitor-query-readme\">Java<\/a>, <a href=\"https:\/\/learn.microsoft.com\/javascript\/api\/overview\/azure\/monitor-query-readme\">JavaScript<\/a>, and <a href=\"https:\/\/learn.microsoft.com\/python\/api\/overview\/azure\/monitor-query-readme\">Python<\/a>. As adoption of the libraries increased, customer feedback indicated demand for a Go library. In the interest of being customer-driven, the team shipped the <a href=\"https:\/\/pkg.go.dev\/github.com\/Azure\/azure-sdk-for-go\/sdk\/monitor\/azquery\">Go<\/a> library&#8217;s <a href=\"https:\/\/devblogs.microsoft.com\/azure-sdk\/announcing-the-stable-release-of-the-azure-monitor-query-client-module-for-go\/\">initial stable release<\/a> in February 2023.<\/p>\n<p>Beyond requests to support extra programming languages, customer feedback also provided a strong signal on shortcomings of the existing log query APIs. The libraries restricted log queries to the <em>workspace-centric<\/em> style\u2014read-only Kusto queries that target one or more Log Analytics workspaces by workspace ID. As explained on GitHub in <a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-net\/issues\/26638\">this .NET issue<\/a> and <a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-python\/issues\/20417\">this Python issue<\/a>, there are situations in which this query style doesn&#8217;t suffice. To address this problem, <em>resource-centric<\/em> log query APIs are needed. Resource-centric queries target logs for a specific Azure resource by its resource ID.<\/p>\n<p>Resource-centric log query APIs are available as of the following package versions:<\/p>\n<table>\n<thead>\n<tr>\n<th>Ecosystem<\/th>\n<th>Minimum package version<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>.NET<\/td>\n<td><a href=\"https:\/\/www.nuget.org\/packages\/Azure.Monitor.Query\/1.2.0\">1.2.0<\/a><\/td>\n<\/tr>\n<tr>\n<td>Go<\/td>\n<td><a href=\"https:\/\/pkg.go.dev\/github.com\/Azure\/azure-sdk-for-go\/sdk\/monitor\/azquery@v1.1.0\">1.1.0<\/a><\/td>\n<\/tr>\n<tr>\n<td>Java<\/td>\n<td><a href=\"https:\/\/central.sonatype.com\/artifact\/com.azure\/azure-monitor-query\/1.2.0\/overview\">1.2.0<\/a><\/td>\n<\/tr>\n<tr>\n<td>JavaScript<\/td>\n<td><a href=\"https:\/\/www.npmjs.com\/package\/@azure\/monitor-query\/v\/1.1.0\">1.1.0<\/a><\/td>\n<\/tr>\n<tr>\n<td>Python<\/td>\n<td><a href=\"https:\/\/pypi.org\/project\/azure-monitor-query\/1.2.0\/\">1.2.0<\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>For demonstration purposes, the .NET library is used for code samples in this blog post.<\/p>\n<h2>Pitfalls with workspace-centric log queries<\/h2>\n<p>Imagine a scenario in which you need to query incoming HTTP request logs for a specific Azure App Service resource. There are a couple things to note:<\/p>\n<ul>\n<li>A Log Analytics workspace may contain data for multiple Azure resources. To handle that situation, you can filter by the <a href=\"https:\/\/learn.microsoft.com\/azure\/azure-monitor\/logs\/log-standard-columns#_resourceid\">standard <code>_ResourceId<\/code> column<\/a> in your Kusto query. For example:\n<pre><code class=\"language-csharp\">var resourceId = _configuration[\"LogsQuery:ResourceId\"]!;\r\nvar workspaceId = _configuration[\"LogsQuery:WorkspaceId\"]!;\r\nvar query = \r\n    $@\"AppServiceHTTPLogs\r\n    | where _ResourceId =~ \"\"{resourceId}\"\"\r\n    | top 5 by TimeGenerated\r\n    | project ComputerName, Result\";\r\n\r\nDateTimeOffset.TryParse(\"5\/24\/2023 03:00:00 PM +00:00\", out var timeStart);\r\nDateTimeOffset.TryParse(\"5\/24\/2023 06:00:00 PM +00:00\", out var timeEnd);\r\n\r\nResponse&lt;IReadOnlyList&lt;HttpLogEntry&gt;&gt; result = \r\n    await _logsQueryClient.QueryWorkspaceAsync&lt;HttpLogEntry&gt;(\r\n        workspaceId,\r\n        query,\r\n        new QueryTimeRange(timeStart, timeEnd));<\/code><\/pre>\n<\/li>\n<li>An Azure resource may send data to multiple Log Analytics workspaces. To handle that situation, you can execute the same Kusto query across multiple workspaces. For example:\n<pre><code class=\"language-csharp\">var resourceId = _configuration[\"LogsQuery:ResourceId\"]!;\r\nvar workspaceId1 = _configuration[\"LogsQuery:WorkspaceId1\"]!;\r\nvar workspaceId2 = _configuration[\"LogsQuery:WorkspaceId2\"]!;\r\nvar workspaceId3 = _configuration[\"LogsQuery:WorkspaceId3\"]!;\r\nvar query = \r\n    $@\"AppServiceHTTPLogs\r\n    | where _ResourceId =~ \"\"{resourceId}\"\"\r\n    | top 5 by TimeGenerated\r\n    | project ComputerName, Result\";\r\n\r\nDateTimeOffset.TryParse(\"5\/24\/2023 03:00:00 PM +00:00\", out var timeStart);\r\nDateTimeOffset.TryParse(\"5\/24\/2023 06:00:00 PM +00:00\", out var timeEnd);\r\n\r\nResponse&lt;IReadOnlyList&lt;HttpLogEntry&gt;&gt; result =\r\n    await _logsQueryClient.QueryWorkspaceAsync&lt;HttpLogEntry&gt;(\r\n        workspaceId1,\r\n        query,\r\n        new QueryTimeRange(timeStart, timeEnd),\r\n        new LogsQueryOptions\r\n        {\r\n            AdditionalWorkspaces = { workspaceId2, workspaceId3 }\r\n        });<\/code><\/pre>\n<\/li>\n<\/ul>\n<p>In many applications, access control restrictions prevent the aforementioned solutions from working. A Log Analytics administrator can query the entire workspace. However, an Azure resource owner can only query logs for their own resources. For more information, see <a href=\"https:\/\/learn.microsoft.com\/azure\/azure-monitor\/logs\/manage-access\">Manage access to Log Analytics workspaces<\/a>. When this workspace-centric query approach doesn&#8217;t suffice, consider a resource-centric query instead.<\/p>\n<h2>A primer on resource-centric log queries<\/h2>\n<p>The following example demonstrates using the .NET library in an ASP.NET Core 7.0 <a href=\"https:\/\/learn.microsoft.com\/aspnet\/core\/razor-pages\/?view=aspnetcore-7.0\">Razor Pages<\/a> web app. Complete the following steps to query a single Azure resource&#8217;s logs:<\/p>\n<ol>\n<li>In the <em>Program.cs<\/em> file, invoke the <a href=\"https:\/\/learn.microsoft.com\/dotnet\/api\/microsoft.extensions.azure.logsqueryclientbuilderextensions.addlogsqueryclient\">AddLogsQueryClient<\/a> and <a href=\"https:\/\/learn.microsoft.com\/dotnet\/api\/microsoft.extensions.azure.azureclientfactorybuilder.usecredential\">UseCredential<\/a> methods:\n<pre><code class=\"language-csharp\">using Azure.Identity;\r\nusing Microsoft.Extensions.Azure;\r\n\r\nvar builder = WebApplication.CreateBuilder(args);\r\n\r\nbuilder.Services.AddRazorPages();\r\nbuilder.Services.AddAzureClients(clientBuilder =&gt;\r\n{\r\n    clientBuilder.AddLogsQueryClient();\r\n    clientBuilder.UseCredential(new DefaultAzureCredential());\r\n});\r\n\r\nvar app = builder.Build();\r\n\r\n\/\/ code omitted for brevity<\/code><\/pre>\n<blockquote><p>NOTE: The <code>AddLogsQueryClient<\/code> method used in the preceding code sample is a new feature as of the 1.2.0 version.<\/p><\/blockquote>\n<p>In the preceding code:<\/p>\n<ul>\n<li>The <code>AddLogsQueryClient<\/code> extension method registers a <a href=\"https:\/\/learn.microsoft.com\/dotnet\/api\/azure.monitor.query.logsqueryclient\">LogsQueryClient<\/a> instance with the dependency injection container.<\/li>\n<li>The <code>UseCredential<\/code> method sets the Azure Active Directory token-based credential type to use when creating the <code>LogsQueryClient<\/code>. That credential type is <code>DefaultAzureCredential<\/code> from the Azure Identity client library.<\/li>\n<\/ul>\n<\/li>\n<li>Query the resource via a call to <a href=\"https:\/\/learn.microsoft.com\/dotnet\/api\/azure.monitor.query.logsqueryclient.queryresource\">QueryResource<\/a> or <a href=\"https:\/\/learn.microsoft.com\/dotnet\/api\/azure.monitor.query.logsqueryclient.queryresourceasync\">QueryResourceAsync<\/a>:\n<pre><code class=\"language-csharp\">using Azure;\r\nusing Azure.Core;\r\nusing Azure.Monitor.Query;\r\nusing Microsoft.AspNetCore.Mvc.RazorPages;\r\n\r\npublic class IndexModel : PageModel\r\n{\r\n    private readonly IConfiguration _configuration;\r\n    private readonly LogsQueryClient _logsQueryClient;\r\n\r\n    public IndexModel(\r\n        IConfiguration configuration,\r\n        LogsQueryClient logsQueryClient)\r\n    {\r\n        _configuration = configuration;\r\n        _logsQueryClient = logsQueryClient;\r\n    }\r\n\r\n    public async Task OnGet()\r\n    {\r\n        var resourceId = _configuration[\"LogsQuery:ResourceId\"]!;\r\n        var query = \r\n            @\"AppServiceHTTPLogs\r\n            | top 5 by TimeGenerated\r\n            | project ComputerName, Result\";\r\n\r\n        DateTimeOffset.TryParse(\"5\/24\/2023 03:00:00 PM +00:00\", out var timeStart);\r\n        DateTimeOffset.TryParse(\"5\/24\/2023 06:00:00 PM +00:00\", out var timeEnd);\r\n\r\n        Response&lt;IReadOnlyList&lt;HttpLogEntry&gt;&gt; result = \r\n            await _client.QueryResourceAsync&lt;HttpLogEntry&gt;(\r\n                new ResourceIdentifier(resourceId),\r\n                query,\r\n                new QueryTimeRange(timeStart, timeEnd));\r\n\r\n        foreach (HttpLogEntry entry in result.Value)\r\n        {\r\n            Console.WriteLine($\"{entry.ComputerName}: {entry.Success}\");\r\n        }\r\n\r\n        \/\/ code omitted for brevity\r\n    }\r\n}\r\n\r\npublic class HttpLogEntry\r\n{\r\n    public required string ComputerName { get; set; }\r\n    public required string Success { get; set; }\r\n}<\/code><\/pre>\n<p>In the preceding code, the <code>LogsQuery:ResourceId<\/code> key is used to fetch the Azure resource ID from <em>appsettings.json<\/em>:<\/p>\n<pre><code class=\"language-json\">{\r\n  \"LogsQuery\": {\r\n    \"ResourceId\": \"\/subscriptions\/&lt;subscription_id&gt;\/resourceGroups\/&lt;resource_group_name&gt;\/providers\/Microsoft.Web\/sites\/&lt;web_app_name&gt;\"\r\n  },\r\n  \"Logging\": {\r\n    \"LogLevel\": {\r\n      \"Default\": \"Information\",\r\n      \"Microsoft.AspNetCore\": \"Warning\"\r\n    }\r\n  },\r\n  \"AllowedHosts\": \"*\"\r\n}<\/code><\/pre>\n<p>To find that resource ID:<\/p>\n<ol>\n<li>Navigate to your resource&#8217;s page in the Azure portal.<\/li>\n<li>From the Overview blade, select the <strong>JSON View<\/strong> link.<\/li>\n<li>In the resulting window, copy the value in the <strong>Resource ID<\/strong> text box.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<h2>Summary<\/h2>\n<p>With support for both workspace- and resource-centric log queries, the Azure Monitor Query libraries provide more flexibility than ever before. Programmatically fetch your log data with ease to gain insights into what&#8217;s happening in your Azure applications.<\/p>\n<p>For language-specific code samples demonstrating resource-centric log queries, see the following links:<\/p>\n<ul>\n<li><a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-net\/tree\/main\/sdk\/monitor\/Azure.Monitor.Query#logs-query\">.NET<\/a><\/li>\n<li><a href=\"https:\/\/pkg.go.dev\/github.com\/Azure\/azure-sdk-for-go\/sdk\/monitor\/azquery#example-LogsClient.QueryResource\">Go<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-java\/blob\/main\/sdk\/monitor\/azure-monitor-query\/README.md#query-logs-by-resource-id\">Java<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-js\/blob\/ceddbec477517d4bd335ee9c65ad835c986134a5\/sdk\/monitor\/monitor-query\/test\/public\/logsQueryClient.spec.ts#L310-L317\">JavaScript<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-python\/blob\/main\/sdk\/monitor\/azure-monitor-query\/samples\/sample_resource_logs_query.py\">Python<\/a><\/li>\n<\/ul>\n<p>Any feedback you have on how we can improve the libraries is greatly appreciated. Let&#8217;s have those conversations on GitHub at these locations:<\/p>\n<ul>\n<li><a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-net\/issues\">.NET<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-go\/issues\">Go<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-java\/issues\">Java<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-js\/issues\">JavaScript<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-python\/issues\">Python<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Learn how to query logs for specific Azure resources with the Azure Monitor Query libraries.<\/p>\n","protected":false},"author":63456,"featured_media":2701,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[804,24],"class_list":["post-2700","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure-sdk","tag-azure-monitor","tag-releases"],"acf":[],"blog_post_summary":"<p>Learn how to query logs for specific Azure resources with the Azure Monitor Query libraries.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts\/2700","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/users\/63456"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/comments?post=2700"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts\/2700\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/media\/2701"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/media?parent=2700"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/categories?post=2700"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/tags?post=2700"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}