If you’re running a production application, you know the challenges of managing logs. Too few logs and you’re flying blind; too many and you’re drowning in data and paying excessive storage costs. It’s a classic observability dilemma – you want comprehensive information when things go wrong, but you don’t want to store every detail from your happy paths.<\/p>\n
Enter log sampling<\/a> in .NET – a powerful capability that lets you strategically reduce log volume while maintaining observability. Unlike simple log filtering which uses binary decisions (emit or don’t emit logs), sampling gives you fine-grained control, letting you emit a precise percentage of logs from different parts of your application.<\/p>\n To get started, install the \ud83d\udce6 Microsoft.Extensions.Telemetry<\/a> NuGet package:<\/p>\n or add it directly in the C# project file:<\/p>\n As applications grow in complexity and traffic, logging strategies that worked well during development can quickly become problematic in production environments. Consider these common scenarios:<\/p>\n Traditional approaches to these problems often involve simply turning off lower-level logs in production – but this creates blind spots. What if you could instead keep the same logging instrumentation but intelligently sample those logs?<\/p>\n Log sampling extends filtering capabilities<\/a> by giving you more fine-grained control over which logs are emitted. Instead of enabling or disabling entire categories of logs, sampling lets you emit a controlled percentage of them.<\/p>\n For example, while filtering typically uses probabilities like .NET provides several sampling strategies out of the box:<\/p>\n Let’s walk through how to implement each approach.<\/p>\n Random probabilistic sampling is usually the easiest place to start. It allows you to define sampling rules based on log category, log level, or event ID.<\/p>\n For basic scenarios, you can configure a single probability value:<\/p>\n This configuration samples 1% of Information logs.<\/p>\n For more complex scenarios, you can define specific sampling rules:<\/p>\n Random probabilistic sampling supports runtime configuration updates via the <xref:Microsoft.Extensions.Options.IOptionsMonitor%601> interface. If you’re using a configuration provider that supports reloads\u2014such as the File Configuration Provider<\/a> – you can update sampling rules at runtime without restarting the application. For example, depending on how your application is currently behaving in production, you can increase sampling rate or even enable sampling for all logs in case you need more logs to help with diagnostics. After diagnostics and troubleshooting, you can revert to the original sampling rate – again, all without restarting the application!<\/p>\n When using distributed tracing, it’s often valuable to ensure that logs and traces are sampled consistently. Trace-based sampling ensures that logs are only emitted if the underlying trace is being recorded:<\/p>\n This approach is particularly useful in microservice architectures where you want to maintain correlation between traces and logs.<\/p>\n For specialized requirements, you can create a custom sampler by deriving from the abstract class.\nFor example, this is how a rate limiting sampler allowing no more than 1 log record per second might look like:<\/p>\n Register it:<\/p>\n And voil\u00e0! You have a custom log sampler that allows only one log record per second!<\/p>\n The sampling logic introduces a minimal amount of CPU overhead in the application. However, whenever a specific log record is eliminated, it saves CPU cycles, transmission costs, storage costs.\nWe expect that for most situations, the savings or costs will be in the noise. But as always for performance issues, it’s always safer to measure and ensure there are no negative effects.<\/p>\n Log sampling gives you a powerful middle path between capturing every log (expensive) and capturing only high-level logs (insufficient detail). By strategically reducing the volume of routine logs while ensuring critical information is always captured, you can significantly reduce storage costs without sacrificing observability.<\/p>\n Whether you use random sampling, trace-based sampling, or a custom implementation, the ability to fine-tune your logging volume is a valuable tool in your observability toolkit.<\/p>\n Explores the new log sampling feature<\/p>\n","protected":false},"author":190322,"featured_media":56826,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[685,7699],"tags":[7701,7530],"class_list":["post-56825","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dotnet","category-dotnet-fundamentals","tag-dotnet-8","tag-logging"],"acf":[],"blog_post_summary":" Explores the new log sampling feature<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/56825","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/users\/190322"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/comments?post=56825"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/56825\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media\/56826"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media?parent=56825"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/categories?post=56825"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/tags?post=56825"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Get started<\/h2>\n
dotnet add package Microsoft.Extensions.Telemetry<\/code><\/pre>\n<ItemGroup>\n <PackageReference Include=\"Microsoft.Extensions.Telemetry\"\n Version=\"*\" \/>\n<\/ItemGroup><\/code><\/pre>\nThe challenge of logging at scale<\/h2>\n
\n
Log sampling to the rescue<\/h2>\n
0<\/code> (emit no logs) or 1<\/code> (emit all logs), sampling lets you choose any value in between, such as 0.1<\/code> to emit 10% of logs, or 0.25<\/code> to emit 25%.<\/p>\n\n
Random probabilistic sampling: The simplest approach<\/h3>\n
builder.Logging.AddRandomProbabilisticSampler(0.01, LogLevel.Information);<\/code><\/pre>\nbuilder.Logging.AddRandomProbabilisticSampler(options =>\n{\n \/\/ Sample 5% of logs with event ID 1001 across all categories\n options.Rules.Add(\n new RandomProbabilisticSamplerFilterRule(\n probability: 0.05d,\n eventId : 1001));\n});<\/code><\/pre>\nChange the sampling rate dynamically<\/h4>\n
Trace-based sampling: Keeping logs and traces in sync<\/h3>\n
builder.Logging.AddTraceBasedSampler();<\/code><\/pre>\nGoing fancy: Implementing your own sampling strategy<\/h3>\n
internal sealed class RateLimitingSampler : LoggingSampler\n{\n private readonly Stopwatch stopwatch = Stopwatch.StartNew();\n private readonly long maxBalance = Stopwatch.Frequency;\n private long currentBalance;\n\n public RateLimitingSampler()\n {\n currentBalance = stopwatch.ElapsedTicks - maxBalance;\n }\n public override bool ShouldSample<TState>(in LogEntry<TState> logEntry)\n {\n long currentTicks;\n long currentBalanceTicks;\n long availableBalanceAfterWithdrawal;\n do\n {\n currentBalanceTicks = Interlocked.Read(ref currentBalance);\n currentTicks = stopwatch.ElapsedTicks;\n var currentAvailableBalance = currentTicks - currentBalanceTicks;\n if (currentAvailableBalance > maxBalance)\n {\n currentAvailableBalance = maxBalance;\n }\n\n availableBalanceAfterWithdrawal = currentAvailableBalance - Stopwatch.Frequency;\n if (availableBalanceAfterWithdrawal < 0)\n {\n return false;\n }\n }\n\n while (Interlocked.CompareExchange(ref currentBalance, currentTicks - availableBalanceAfterWithdrawal, currentBalanceTicks) != currentBalanceTicks);\n return true;\n }\n}<\/code><\/pre>\nbuilder.Logging..AddSampler<RateLimitingSampler>();<\/code><\/pre>\nPerformance considerations<\/h2>\n
Summary<\/h2>\n
References<\/h2>\n
\n