{"id":1778,"date":"2022-01-13T10:42:56","date_gmt":"2022-01-13T18:42:56","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/azure-sdk\/?p=1778"},"modified":"2022-01-13T14:05:07","modified_gmt":"2022-01-13T22:05:07","slug":"distributed-tracing-with-azure-functions-event-grid-triggers","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/azure-sdk\/distributed-tracing-with-azure-functions-event-grid-triggers\/","title":{"rendered":"Distributed tracing with Azure Functions Event Grid triggers"},"content":{"rendered":"<p>The Azure Event Grid client libraries support distributed tracing for the CloudEvents schema. They populate the <a href=\"https:\/\/github.com\/cloudevents\/spec\/blob\/v1.0.1\/extensions\/distributed-tracing.md\">Distributed Tracing extension<\/a> that allows connecting event consumer telemetry to producer calls. The <a href=\"https:\/\/docs.microsoft.com\/azure\/event-grid\/troubleshoot-issues#distributed-tracing\">Event Grid documentation<\/a> shows how to enable tracing in the producer. It also shows how to configure the Event Hubs or Service Bus subscription.<\/p>\n<p>Azure Functions supports <a href=\"https:\/\/docs.microsoft.com\/azure\/azure-monitor\/app\/azure-functions-supported-features\">distributed tracing with Azure Monitor<\/a>, which includes built-in tracing of executions and bindings, performance monitoring, and more.<\/p>\n<p><a href=\"https:\/\/www.nuget.org\/packages\/Microsoft.Azure.WebJobs.Extensions.EventGrid\/\">Microsoft.Azure.WebJobs.Extensions.EventGrid<\/a> package version 3.1.0 or later enables correlation for CloudEvents between producer calls and Functions Event Grid trigger executions as shown on the screenshot below.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-content\/uploads\/sites\/58\/2022\/01\/2022-01-19-cloudevent_dotnet_default.png\" alt=\"Azure portal screenshot showing Azure Function consumer calls linked to producer\" \/><\/p>\n<p>If you use an Event Grid trigger with the CloudEvents schema, you only need to update the <code>Microsoft.Azure.WebJobs.Extensions.EventGrid<\/code> package to the latest version. In other cases, Functions still trace trigger calls so you can monitor how events are processed. However, the connection between send calls on producer and Function trigger calls is missing in Azure Monitor&#8217;s end-to-end trace.<\/p>\n<p>In this post, I&#8217;ll show how to enable correlation in .NET and Java Functions for:<\/p>\n<ul>\n<li><a href=\"https:\/\/docs.microsoft.com\/azure\/event-grid\/event-schema\">Event Grid<\/a> schema with Event Grid trigger<\/li>\n<li><a href=\"https:\/\/cloudevents.io\/\">CloudEvents<\/a> schema with HTTP trigger (webhook)<\/li>\n<\/ul>\n<h2>Get started<\/h2>\n<p>Before enabling correlation, make sure you get telemetry from producer and Functions in Azure Monitor:<\/p>\n<ol>\n<li>\n<p>If you publish CloudEvents using Azure Event Grid client library version 4 with <a href=\"https:\/\/docs.microsoft.com\/azure\/event-grid\/troubleshoot-issues#distributed-tracing\">distributed tracing enabled<\/a>, it:<\/p>\n<ul>\n<li>Populates <a href=\"https:\/\/github.com\/cloudevents\/spec\/blob\/v1.0.1\/extensions\/distributed-tracing.md\">Distributed Tracing extension<\/a> on the events.<\/li>\n<li>Traces outgoing calls<\/li>\n<\/ul>\n<\/li>\n<li>\n<p>If you use another library to publish events and it doesn&#8217;t support tracing, see the example of manual instrumentation for Event Grid schema below.<\/p>\n<\/li>\n<li>\n<p>Enable <a href=\"https:\/\/docs.microsoft.com\/azure\/azure-monitor\/app\/monitor-functions\">tracing on Azure Functions<\/a>.<\/p>\n<\/li>\n<li>\n<p>If you use Event Grid trigger and CloudEvents schema in your Functions, update the <a href=\"https:\/\/www.nuget.org\/packages\/Microsoft.Azure.WebJobs.Extensions.EventGrid\">Microsoft.Azure.WebJobs.Extensions.EventGrid<\/a> package to version 3.1.0. You don&#8217;t need to change your Functions code to enable correlation.<\/p>\n<\/li>\n<\/ol>\n<h2>Connect send and Function calls<\/h2>\n<p>To correlate Azure Function executions to producer traces, we&#8217;re going to read context from the event and populate it on Azure Functions telemetry. For CloudEvents, we&#8217;ll just read it from the Distributed Tracing extension. For the Event Grid schema, we&#8217;ll have to read and write custom properties in the <code>data<\/code> payload.<\/p>\n<h3>.NET<\/h3>\n<p>Azure Functions uses <a href=\"https:\/\/docs.microsoft.com\/dotnet\/core\/diagnostics\/distributed-tracing-instrumentation-walkthroughs\">System.Diagnostics.Activity<\/a> under-the-hood. We&#8217;ll update the <code>Activity<\/code> created by Functions to <em>link<\/em> the producer context. Links connect independent traces together in the Azure Monitor UX.<\/p>\n<h4>CloudEvents with webhook (HTTP) trigger<\/h4>\n<ol>\n<li>\n<p>Here, we extract the context from the CloudEvent and link it to the <code>Activity<\/code>. It&#8217;s done in the Azure Monitor-specific format:<\/p>\n<pre><code class=\"language-csharp\">using System.Collections.Generic;\r\nusing System.Diagnostics;\r\nusing System.Text.Json;\r\n\/\/ code omitted for brevity\r\n\r\npublic static void LinkContext(this Activity activity, \r\n    IDictionary&lt;string, object&gt; extensionAttributes)\r\n{\r\n   if (activity != null &amp;&amp;\r\n       extensionAttributes.TryGetValue(\"traceparent\", out var tp) &amp;&amp;\r\n       tp is string traceparent &amp;&amp;\r\n       IsValidTraceparent(traceparent))\r\n   {\r\n       var link = new AzureMonitorLink\r\n       (\r\n           \/\/ parse traceparent according to https:\/\/www.w3.org\/TR\/trace-context\/\r\n           traceparent.Substring(3, 32), \/\/ traceId\r\n           traceparent.Substring(36, 16) \/\/ spanId\r\n       );\r\n\r\n       \/\/ consider formatting JSON manually for best performance\r\n       activity.AddTag(\"_MS.links\", JsonSerializer.Serialize(new[] { link }));\r\n\r\n       if (extensionAttributes.TryGetValue(\"tracestate\", out var ts) &amp;&amp;\r\n           ts is string tracestate)\r\n       {\r\n           activity.TraceStateString = tracestate;\r\n       }\r\n   }\r\n}\r\n\r\nprivate static bool IsValidTraceparent(string traceparent) =&gt; traceparent != null &amp;&amp; \r\n   traceparent.StartsWith(\"00-\") &amp;&amp; traceparent.Length == 55;\r\n\r\n\/\/ Property names match Azure Monitor's over-the-wire format. Don't change them.\r\nprivate readonly record struct AzureMonitorLink(string operation_Id, string id);<\/code><\/pre>\n<\/li>\n<li>\n<p>Now we need to call the <code>LinkContext<\/code> method in the Function execution as early as possible.<\/p>\n<pre><code class=\"language-csharp\">using Azure.Messaging;\r\nusing System.Diagnostics;\r\n\/\/ code omitted for brevity\r\n\r\n[FunctionName(\"MyFunction\")]\r\npublic static async Task&lt;IActionResult&gt; RunAsync(\r\n        [HttpTrigger(AuthorizationLevel.Anonymous, \"POST\", \"OPTIONS\", Route = \"handler\")] HttpRequest req,\r\n        ILogger log)\r\n{\r\n   \/\/ handshake, code omitted for brevity\r\n\r\n   var @event = CloudEvent.Parse(BinaryData.FromStream(req.Body));\r\n\r\n   \/\/ Activity can be null if Azure Monitor isn't enabled.\r\n   Activity.Current?.LinkContext(@event.ExtensionAttributes);\r\n   \/\/ ...\r\n}<\/code><\/pre>\n<\/li>\n<li>\n<p>Deploy your Function and trigger it. It takes up to 5 minutes for data to propagate and become accessible through the Azure portal.<\/p>\n<\/li>\n<li>\n<p>View your traces in the Azure portal by navigating to <strong>Transaction search<\/strong> on the left for your Application Insights resource. Select <strong>See all data in the last 24 hours<\/strong>.<\/p>\n<\/li>\n<\/ol>\n<p>The following Azure portal screenshot shows Azure Function consumer calls linked to the producer:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-content\/uploads\/sites\/58\/2022\/01\/2022-01-19-cloudevent_dotnet.png\" alt=\"Azure portal screenshot showing Azure Function consumer calls linked to producer\" \/><\/p>\n<h4>CloudEvents: batch processing<\/h4>\n<p>With batching enabled on Event Grid subscription, we&#8217;ll receive multiple events at once and link each of them.<\/p>\n<ol>\n<li>\n<p>Let&#8217;s add a <code>LinkContext<\/code> method that populates several links to <code>Activity<\/code>. Adding multiple <code>tracestate<\/code> properties is unsupported, but it doesn&#8217;t affect correlation between producer and Function.<\/p>\n<pre><code class=\"language-csharp\">using Azure.Messaging;\r\nusing System.Collections.Generic;\r\nusing System.Diagnostics;\r\nusing System.Linq;\r\nusing System.Text.Json;\r\n\/\/ code omitted for brevity\r\n\r\npublic static void LinkContext(this Activity activity, IEnumerable&lt;CloudEvent&gt; events)\r\n{\r\n   if (activity != null &amp;&amp; events.Any())\r\n   {\r\n       var links = new List&lt;AzureMonitorLink&gt;();\r\n       foreach (CloudEvent @event in events)\r\n       {\r\n           if (@event.ExtensionAttributes.TryGetValue(\"traceparent\", out var tp) &amp;&amp; tp is string traceparent &amp;&amp;\r\n               IsValidTraceparent(traceparent))\r\n           {\r\n               links.Add(new AzureMonitorLink(\r\n                           traceparent.Substring(3, 32), \/\/ traceId\r\n                           traceparent.Substring(36, 16))); \/\/ spanId\r\n\r\n               \/\/ multiple tracestates are not currently supported.\r\n           }\r\n       }\r\n\r\n       activity.AddTag(\"_MS.links\", JsonSerializer.Serialize(links));\r\n   }\r\n}<\/code><\/pre>\n<\/li>\n<li>\n<p>Now we need to call the <code>LinkContext<\/code> method in Function execution as soon as events are deserialized.<\/p>\n<pre><code class=\"language-csharp\">using Azure.Messaging;\r\nusing System.Diagnostics;\r\n\/\/ code omitted for brevity\r\n\r\n[FunctionName(\"BatchWebhook\")]\r\npublic static async Task&lt;IActionResult&gt; RunAsync(\r\n        [HttpTrigger(AuthorizationLevel.Anonymous, \"POST\", \"OPTIONS\", Route = \"handler\")] HttpRequest req,\r\n        ILogger log)\r\n{\r\n   \/\/ handshake, code omitted for brevity\r\n\r\n   var events = CloudEvent.ParseMany(BinaryData.FromStream(req.Body));\r\n   Activity.Current?.LinkContext(events);\r\n   \/\/ ...\r\n}<\/code><\/pre>\n<\/li>\n<\/ol>\n<h4>Event Grid schema<\/h4>\n<p>The Event Grid schema doesn&#8217;t have dedicated properties to propagate trace context. We&#8217;ll use custom properties, inject them into the event on the producer side, and then read them in the Function. We&#8217;ll use the same approach to link producer trace context to <code>Activity<\/code> created by Azure Functions.<\/p>\n<ol>\n<li>\n<p>Update your Event Grid data model definition to include <code>traceparent<\/code> and <code>tracestate<\/code> properties:<\/p>\n<pre><code class=\"language-csharp\">using System.Text.Json.Serialization;\r\n\/\/ code omitted for brevity\r\n\r\ninternal readonly record struct EventGridData\r\n{\r\n    [JsonPropertyName(\"traceparent\")]\r\n    public string Traceparent { get; init; }\r\n\r\n    [JsonPropertyName(\"tracestate\")]\r\n    public string Tracestate { get; init; }\r\n\r\n    \/\/ code omitted for brevity\r\n}<\/code><\/pre>\n<\/li>\n<li>\n<p>On the producer side, we&#8217;ll create a new <code>Activity<\/code> and add <code>traceparent<\/code> and <code>tracestate<\/code> to event data.<\/p>\n<ul>\n<li>\n<p>If you use the <a href=\"https:\/\/docs.microsoft.com\/azure\/azure-monitor\/app\/app-insights-overview\">Application Insights SDK<\/a>, track the new <code>DependencyTelemetry<\/code> using the <code>StartOperation<\/code> method. It will create a new <code>Activity<\/code> under-the-hood. Inject the context of the new <code>Activity<\/code> to the event data.<\/p>\n<pre><code class=\"language-csharp\">using Azure.Messaging.EventGrid;\r\nusing Microsoft.ApplicationInsights;\r\nusing Microsoft.ApplicationInsights.DataContracts;\r\nusing System.Diagnostics;\r\n\/\/ code omitted for brevity\r\n\r\nusing (var sendEventDependency = \r\n    telemetryClient.StartOperation&lt;DependencyTelemetry&gt;(\"Send Event Grid event\"))\r\n{\r\n sendEventDependency.Telemetry.Type = \"InProc\";\r\n var eventData = new EventGridData\r\n {\r\n     Traceparent = Activity.Current.Id,\r\n     Tracestate = Activity.Current.TraceStateString\r\n };\r\n\r\n var @event = new EventGridEvent(\"subject\", \"type\", \"data-version\", eventData);\r\n await eventCollector.AddAsync(@event);\r\n}<\/code><\/pre>\n<\/li>\n<li>\n<p>If you use <a href=\"https:\/\/devblogs.microsoft.com\/azure-sdk\/introducing-experimental-opentelemetry-support-in-the-azure-sdk-for-net\/\">OpenTelemetry (experimental support)<\/a>, create a new <code>Activity<\/code> using custom <a href=\"https:\/\/docs.microsoft.com\/dotnet\/api\/system.diagnostics.activitysource\">ActivitySource<\/a>. For more information on using <code>ActivitySource<\/code>, see <a href=\"https:\/\/docs.microsoft.com\/dotnet\/core\/diagnostics\/distributed-tracing-instrumentation-walkthroughs\">Adding distributed tracing instrumentation<\/a>. As a note, <code>Activity<\/code> can be <code>null<\/code> here.<\/p>\n<pre><code class=\"language-csharp\">using Azure.Messaging.EventGrid;\r\nusing System.Diagnostics;\r\n\/\/ code omitted for brevity\r\n\r\n\/\/ make sure to enable this ActivitySource when configuring OpenTelemetry\r\nprivate static ActivitySource source = new ActivitySource(\"MyEventGridProducer\");\r\n\/\/ code omitted for brevity\r\n\r\nusing (var sendActivity = source.StartActivity(\"Send Event Grid event\"))\r\n{\r\n var eventData = new EventGridData\r\n {\r\n     Traceparent = sendActivity?.Id,\r\n     Tracestate = sendActivity?.TraceStateString\r\n };\r\n\r\n var @event = new EventGridEvent(\"subject\", \"type\", \"data-version\", eventData);\r\n await publisherClient.SendEventAsync(@event);\r\n}<\/code><\/pre>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p>Azure Functions consumer changes are similar to the CloudEvents example above. The only difference is how <code>traceparent<\/code> and <code>tracestate<\/code> are obtained from the <code>data<\/code> property. Modify this code to use your data model definition.<\/p>\n<pre><code class=\"language-csharp\">using System.Diagnostics;\r\nusing System.Text.Json;\r\nusing System.Text.Json.Serialization;\r\n\/\/ code omitted for brevity\r\n\r\npublic static void LinkContext(this Activity activity, EventGridData eventData)\r\n{\r\n   if (activity != null &amp;&amp; IsValidTraceparent(eventData.Traceparent))\r\n   { \r\n       var link = new AzureMonitorLink(eventData.Traceparent.Substring(3, 32), \r\n                                       eventData.Traceparent.Substring(36, 16));\r\n\r\n       \/\/ consider formatting JSON manually for best performance\r\n       activity.AddTag(\"_MS.links\", JsonSerializer.Serialize(new[] { link }));\r\n       activity.TraceStateString = eventData.Tracestate;\r\n   }\r\n}<\/code><\/pre>\n<\/li>\n<li>\n<p>Call <code>LinkContext<\/code> method in Function execution as early as possible.<\/p>\n<pre><code class=\"language-csharp\">using Azure.Messaging;\r\nusing System.Diagnostics;\r\n\r\n\/\/ code omitted for brevity\r\n\r\n[FunctionName(\"EventGridFunction\")]\r\npublic void RunEventGrid([EventGridTrigger] EventGridEvent @event, ILogger log)\r\n{\r\n   Activity.Current?.LinkContext(@event.Data.ToObjectFromJson&lt;EventGridData&gt;());\r\n   \/\/ ...\r\n}<\/code><\/pre>\n<\/li>\n<\/ol>\n<p>The following screenshot shows Azure Function consumer calls correlated with producer in the <strong>Transaction diagnostics<\/strong>. In this case, the producer is instrumented with OpenTelemetry:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-content\/uploads\/sites\/58\/2022\/01\/2022-01-19-eventgridevent_dotnet_otel.png\" alt=\"Azure portal screenshot showing Azure Function consumer calls linked to producer\" \/><\/p>\n<h3>Java<\/h3>\n<p>Azure Functions supports distributed tracing for bindings in Java without extra configuration. Azure Monitor <strong>preview<\/strong> support enables collection of custom and rich telemetry from Java Functions. We&#8217;ll need it to correlate Azure Functions and event producer.<\/p>\n<ol>\n<li>\n<p>Enable <a href=\"https:\/\/docs.microsoft.com\/azure\/azure-monitor\/app\/monitor-functions#distributed-tracing-for-java-applications-public-preview\">Azure Monitor for Java Function apps<\/a> (<strong>preview<\/strong>)<\/p>\n<\/li>\n<li>\n<p>Add a dependency on the OpenTelemetry API package: <a href=\"https:\/\/search.maven.org\/artifact\/io.opentelemetry\/opentelemetry-api\">io.opentelemetry:opentelemetry-api<\/a>. For more information, see <a href=\"https:\/\/opentelemetry.io\/docs\/instrumentation\/java\/\">OpenTelemetry documentation<\/a>.<\/p>\n<\/li>\n<li>\n<p>Obtain an OpenTelemetry tracer instance. We&#8217;ll use it to start a new span.<\/p>\n<pre><code class=\"language-java\">import io.opentelemetry.api.GlobalOpenTelemetry;\r\nimport io.opentelemetry.api.trace.Tracer;\r\n\/\/ code omitted for brevity\r\n\r\nprivate final static Tracer TRACER = GlobalOpenTelemetry.getTracer(\"my-function\");<\/code><\/pre>\n<\/li>\n<\/ol>\n<p>The examples below use <code>com.azure.core.models.CloudEvent<\/code> and <code>com.azure.messaging.eventgrid.EventGridEvent<\/code> models from Azure SDKs. You can get them by adding a dependency on <a href=\"https:\/\/search.maven.org\/artifact\/com.azure\/azure-messaging-eventgrid\">com.azure:azure-messaging-eventgrid<\/a>.<\/p>\n<p>If you use different implementations, you might need to adjust these examples for your use case.<\/p>\n<h4>CloudEvents schema<\/h4>\n<p>The <code>Microsoft.Azure.WebJobs.Extensions.EventGrid<\/code> package (version 3.1.0 or later) enables correlation for CloudEvents within Event Grid triggers on Java workers. Check if the <a href=\"https:\/\/docs.microsoft.com\/azure\/azure-functions\/functions-bindings-register#extension-bundles\">extension bundle<\/a> you use includes this support. You may also update <code>Microsoft.Azure.WebJobs.Extensions.EventGrid<\/code> by switching to <a href=\"https:\/\/docs.microsoft.com\/azure\/azure-functions\/functions-bindings-register#explicitly-install-extensions\">explicit extension installation<\/a>; however, using extension bundles is recommended.<\/p>\n<p>If you can&#8217;t update <code>Microsoft.Azure.WebJobs.Extensions.EventGrid<\/code> yet, you can still enable correlation using the following steps:<\/p>\n<ol>\n<li>\n<p>Add a helper class instance that writes the trace context to <code>EventGridEvent<\/code>:<\/p>\n<pre><code class=\"language-java\">import io.opentelemetry.context.propagation.TextMapGetter;\r\n\/\/ code omitted for brevity\r\n\r\nprivate static final Iterable&lt;String&gt; KEYS = List.of(\"traceparent\", \"tracestate\");\r\nprivate static final TextMapGetter&lt;Map&lt;String, Object&gt;&gt; CLOUD_EVENT_GETTER =\r\n        new TextMapGetter&lt;Map&lt;String, Object&gt;&gt;() {\r\n    @Override\r\n    public Iterable&lt;String&gt; keys(Map&lt;String, Object&gt; carrier) { return KEYS; }\r\n\r\n    @Override\r\n    public String get(Map&lt;String, Object&gt; carrier, String key) { \r\n        return carrier.get(key).toString(); \r\n    }\r\n};<\/code><\/pre>\n<\/li>\n<li>\n<p>We&#8217;ll read events from string input here. Since there could be a batch of events, depending upon Event Grid subscription configuration, we&#8217;ll get the trace context from each of them. We can&#8217;t modify telemetry reported by the Azure Functions runtime here. So we&#8217;ll create a new span and link trace contexts from all of the events.<\/p>\n<pre><code class=\"language-java\">import com.azure.core.models.CloudEvent;\r\nimport io.opentelemetry.api.trace.Span;\r\nimport io.opentelemetry.api.trace.SpanBuilder;\r\nimport io.opentelemetry.api.trace.StatusCode;\r\nimport io.opentelemetry.api.trace.propagation.W3CTraceContextPropagator;\r\nimport io.opentelemetry.context.Context;\r\nimport io.opentelemetry.context.Scope;\r\n\/\/ code omitted for brevity\r\n\r\n@FunctionName(\"CloudEvent\")\r\npublic void processCloudEvents(@EventGridTrigger(name=\"eventsStr\") String eventsStr, \r\n           final ExecutionContext context) {\r\n   List&lt;CloudEvent&gt; events = CloudEvent.fromString(eventsStr);\r\n\r\n   SpanBuilder spanBuilder = TRACER.spanBuilder(\"Process CloudEvents\");\r\n\r\n   events.stream().forEach(event -&gt; {\r\n       \/\/ extract trace context from the event using OpenTelemetry propagator.\r\n       Context eventContext = W3CTraceContextPropagator.getInstance().\r\n               extract(Context.current(), event.getExtensionAttributes(), CLOUD_EVENT_GETTER);\r\n\r\n       spanBuilder.addLink(Span.fromContext(eventContext).getSpanContext());\r\n   });\r\n\r\n   Span span = spanBuilder.startSpan();\r\n   try (Scope scope = span.makeCurrent()) {\r\n\r\n       \/\/ process events here\r\n\r\n   } catch (Throwable t) {\r\n       span.setStatus(StatusCode.ERROR);\r\n       throw t;\r\n   } finally {\r\n       span.end();\r\n   }\r\n}<\/code><\/pre>\n<\/li>\n<\/ol>\n<p>If you&#8217;d like to trace each event in the batch separately, you can modify this example to create a span for each event. We&#8217;re rethrowing the exception here. Azure Functions will record it. If you don&#8217;t want to rethrow the exception, you&#8217;ll probably want to record it with <code>span.recordException(ex)<\/code>.<\/p>\n<h4>Event Grid schema<\/h4>\n<p>Similarly to the .NET example, we&#8217;ll use custom properties in event data. Those properties will be populated on the event producer. On the consumer side, we&#8217;ll start a new span and <em>link<\/em> it to the producer trace context.<\/p>\n<ol>\n<li>\n<p>Update your Event Grid data model definition to include <code>traceparent<\/code> and <code>tracestate<\/code> properties:<\/p>\n<pre><code class=\"language-java\">import com.fasterxml.jackson.annotation.JsonProperty;\r\n\/\/ code omitted for brevity\r\n\r\nstatic class EventGridData {\r\n    @JsonProperty(\"traceparent\")\r\n    public String traceparent;\r\n\r\n    @JsonProperty(\"tracestate\")\r\n    public String tracestate;\r\n\r\n    \/\/ code omitted for brevity\r\n}<\/code><\/pre>\n<\/li>\n<li>\n<p>Add a helper class instance that reads trace context from <code>EventGridData<\/code>:<\/p>\n<pre><code class=\"language-java\">import io.opentelemetry.context.propagation.TextMapSetter;\r\n\/\/ code omitted for brevity\r\n\r\nprivate static final TextMapSetter&lt;EventGridData&gt; EVENT_GRID_SETTER =\r\n        new TextMapSetter&lt;EventGridData&gt;() {\r\n    @Override\r\n    public void set(EventGridData carrier, String key, String value) {\r\n        if (\"traceparent\".equals(key)) {\r\n            carrier.traceparent = value;\r\n        } else if (\"tracestate\".equals(key)) {\r\n            carrier.tracestate = value;\r\n        }\r\n    }\r\n};<\/code><\/pre>\n<\/li>\n<li>\n<p>Add <code>traceparent<\/code> and <code>tracestate<\/code> to event data on the producer side:<\/p>\n<pre><code class=\"language-java\">private void sendEventGridEvent() {\r\n   \/\/ change this to your event data model\r\n   EventGridData eventData = new EventGridData();\r\n\r\n   Span span = TRACER.spanBuilder(\"Send Event Grid event\").startSpan();\r\n   try (Scope unused = span.makeCurrent()) {\r\n       \/\/ inject context into EventGridData\r\n       W3CTraceContextPropagator.getInstance().inject(Context.current(), \r\n            eventData, EVENT_GRID_SETTER);\r\n\r\n       eventGridClient.sendEvent(new EventGridEvent(\"subject\", \"type\", \r\n            BinaryData.fromObject(eventData), \"data-version\"));\r\n   } catch (Throwable t) {\r\n       span.setStatus(StatusCode.ERROR);\r\n       throw t;\r\n   } finally {\r\n       span.end();\r\n   }\r\n}<\/code><\/pre>\n<\/li>\n<li>\n<p>Similar to the CloudEvents example, read the trace context from the event data and trace event processing:<\/p>\n<pre><code class=\"language-java\">import com.azure.messaging.eventgrid.EventGridEvent;\r\n\/\/ code omitted for brevity\r\n\r\n@FunctionName(\"EventGridEvent\")\r\npublic void processEventGridEvents(@EventGridTrigger(name=\"eventsStr\") String eventsStr,\r\n            final ExecutionContext context) {\r\n   List&lt;EventGridEvent&gt; events = EventGridEvent.fromString(eventsStr);\r\n\r\n   SpanBuilder spanBuilder = TRACER.spanBuilder(\"Process EventGridEvents\");\r\n\r\n   events.stream().forEach( event -&gt; {\r\n       EventGridData data = event.getData().toObject(EventGridData.class);\r\n       Context eventContext = W3CTraceContextPropagator.getInstance().\r\n               extract(Context.current(), data, EVENT_GRID_GETTER);\r\n\r\n       spanBuilder.addLink(Span.fromContext(eventContext).getSpanContext());\r\n   });\r\n\r\n   Span span = spanBuilder.startSpan();\r\n   try (Scope scope = span.makeCurrent()) {\r\n\r\n       \/\/ process events here\r\n\r\n   } catch (Throwable t) {\r\n       span.setStatus(StatusCode.ERROR);\r\n       throw t;\r\n   } finally {\r\n       span.end();\r\n   }\r\n}\r\n\r\nprivate static final TextMapGetter&lt;EventGridData&gt; EVENT_GRID_GETTER = \r\n       new TextMapGetter&lt;EventGridData&gt;() {\r\n   @Override\r\n   public Iterable&lt;String&gt; keys(EventGridData carrier) { return KEYS; }\r\n\r\n   @Override\r\n   public String get(EventGridData carrier, String key) {\r\n       if (\"traceparent\".equals(key)) {\r\n           return carrier.traceparent;\r\n       } else if (\"tracestate\".equals(key)) {\r\n           return carrier.tracestate;\r\n       }\r\n       return null;\r\n   }\r\n};<\/code><\/pre>\n<\/li>\n<\/ol>\n<p>The following screenshot shows Azure Function consumer calls correlated with the producer in the <strong>Transaction viewer<\/strong>. It shows the case in which batching is configured on the Event Grid subscription. The Functions runtime (with <code>Microsoft.Azure.WebJobs.Extensions.EventGrid<\/code> version 2) tracks a single Function execution that results in three calls to the Java worker.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-content\/uploads\/sites\/58\/2022\/01\/2022-01-19-eventgridevent_java_batching.png\" alt=\"Azure portal screenshot showing Azure Function consumer calls linked to producer\" \/><\/p>\n<h2>Want to hear more?<\/h2>\n<p>Thanks for reading this Azure SDK blog post. What do you think of distributed tracing in the Azure SDK? We&#8217;re actively seeking feedback on this feature, so let us know!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Enable end-to-end distributed tracing for Event Grid triggers on Azure Functions for CloudEvents and Event Grid schema.<\/p>\n","protected":false},"author":75505,"featured_media":1793,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[846,744,757,732,705],"class_list":["post-1778","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure-sdk","tag-distributed-tracing","tag-eventgrid","tag-functions","tag-release","tag-sdk"],"acf":[],"blog_post_summary":"<p>Enable end-to-end distributed tracing for Event Grid triggers on Azure Functions for CloudEvents and Event Grid schema.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts\/1778","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/users\/75505"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/comments?post=1778"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts\/1778\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/media\/1793"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/media?parent=1778"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/categories?post=1778"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/tags?post=1778"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}