January 26th, 2021

How to choose the right Azure Event Hubs .NET client library type for your application

Jesse Squire
Jesse Squire
Senior Software Engineer

Azure Event Hubs is a highly scalable publish-subscribe service that can ingest millions of events per second and stream them to multiple consumers. This lets you process and analyze the massive amounts of data produced by your connected devices and applications. Once Event Hubs has collected the data, you can retrieve, transform, and store it using any real-time analytics provider, such as Azure Stream Analytics, or with batching/storage adapters.

Since its introduction, the Event Hubs service and the libraries used to interact with it have been evolving to offer more features and a world-class experience for developers. As a consequence of continual improvement while avoiding changes that may break existing applications, several generations of libraries for Azure Event Hubs have been released into the ecosystem. In addition to there being multiple libraries, each offers a slightly different set of clients for use in applications. Because of the number of options, it can be confusing to know which is best suited for your application.

This article will focus on the current generation client library, Azure.Messaging.EventHubs, which we recommend for developing of new applications and, where possible when enhancing existing applications. Our goal is that, after reading, you have an understanding of the producer, consumer, and processor types that are offered and the scenarios where they are best applied.

Event Hubs Client Library Packages for .NET

Azure.Messaging.EventHubs

Azure.Messaging.EventHubs is the current generation of the library and is under active development, with improvements and new features being released on a regular cadence. It supports the netstandard2.0 platform, allowing it to be used with a wide range of host environments including .NET 5, .NET Core, and the full .NET Framework. The library retains high-level feature parity with the previous generation, Microsoft.Azure.EventHubs, but with a more discrete client hierarchy and improved API.

The Azure.Messaging.EventHubs library is part of the initiative to improve the development experience across Azure services. To this end, it follows a set of uniform design guidelines intended to drive a consistent experience across development languages and employ established API patterns for all Azure services. The library also follows a set of .NET-specific guidelines to ensure that the .NET SDK has a natural and idiomatic feel that mirrors that of the .NET base class libraries.

The Azure.Messaging.EventHubs library also provides the ability to share in some of the cross-service improvements made to the Azure development experience, such as a unified diagnostics pipeline offering a common view of the activities across each of the client libraries. Another key improvement is a streamlined and simplified experience for authentication using the new Azure.Identity library to share credentials between the clients for different Azure services.

Microsoft.Azure.EventHubs (legacy)

This is the previous generation library, the first to be dedicated to Event Hubs. This library is still supported by Microsoft, and will be well into the future. However, development is focused only on bug fixes and critical updates. It will not be receiving the new features and enhancements developed for Azure.Messaging.EventHubs.

We recommend using this library only for legacy applications that have an existing dependency on it, not for developing new applications.

Windows.Azure.ServiceBus (legacy)

This is the oldest of the legacy libraries, supporting the Azure Messaging offerings as a whole rather than being dedicated to any one service. This design stems from when Event Hubs was still bound to Service Bus and was not a true stand-alone product. The library supports only the full .NET Framework and is not compatible with .NET Core. It offers a different feature set than the more recent client libraries, and does not have high-level feature parity with the newer generations.

While this library is still supported by Microsoft, development is limited to critical bug fixes. It will not be receiving the new features and enhancements developed for Azure.Messaging.EventHubs. We recommend using this library only for legacy applications that have an existing dependency on it, not for developing new applications.

Choosing a Client from Azure.Messaging.EventHubs

I want to… Event Hubs clients More information
Explore Event Hubs by publishing and reading events EventHubProducerClient
and
EventHubConsumerClient 		Sample
Publish events EventHubProducerClient Sample
Read events from all partitions of an Event Hub in a typical production application, using Azure Storage Blobs to store checkpoint and load balancing data EventProcessorClient Sample
Read events from a single partition of an Event Hub EventConsumerClient Sample
Read events from all partitions of an Event Hub and store checkpoint and load balancing data somewhere other than Azure Storage Blobs EventProcessor Article
Read events from a single partition of an Event Hub for an application with high-throughput needs PartitionReceiver Sample
Read events from all partitions of an Event Hub for an application with high-throughput needs EventProcessor Article
Read events from a single partition of an Event Hub using a batch-based approach PartitionReceiver Sample
Read events from all partitions of an Event Hub using a batch-based approach EventProcessor Article

Azure.Messaging.EventHubs Client Details

The Azure.Messaging.EventHubs library has the goal of providing an approachable onboarding experience for developers new to messaging and/or Event Hubs, focusing on enabling a quick initial feedback loop for publishing and consuming events. As developers shift from exploration to tackling real-world production scenarios, the library offers a gradual step-up path, building on the onboarding experience with additional robustness while maintaining a familiar API surface. For applications with high-throughput or special needs, a set of primitives are available to allow developers to work at a low-level and assert more control.

Each Event Hubs client targets a specific area of Event Hubs functionality, presenting an API focused on a concrete set of scenarios. We believe this avoids the distraction of a more complex API surface with areas which are likely to be needed only for more specialized scenarios. To support this design, the client hierarchy has been organized to align with two general categories, mainstream and specialized.

The mainstream set of clients were designed to address the most common production application scenarios and to provide an onboarding experience for those new to Event Hubs. The specialized set of clients focus on high-throughput and enabling developers to assert a higher degree of control. This section will briefly introduce the clients in both categories, though samples will continue to focus heavily on the mainstream clients.

Mainstream

Event Hub Producer Client

The EventHubProducerClient is responsible for publishing event data to the associated Event Hub. The client supports a flexible set of options when publishing, offering a reasonable set of defaults for exploring Event Hubs and for the majority of production scenarios. As application needs become more specialized, the options can be tuned to allow for greater control over publishing and optionally bypassing some of the validation offered by the client for higher throughput.

Event Hub Consumer Client

The EventHubConsumerClient serves as the onboarding point for consuming events by reading from all partitions without the rigor and complexity that you would need in a production application. This is intended to streamline and simplify exploration, allowing developers a feedback loop without the need to understand many of the underlying details.

The production scenario for the EventHubConsumerClient is focused on reading events from a single partition with an iterator-based approach, providing a streaming experience using the await foreach syntax of an asynchronous enumerable. This pattern allow events to be read in a continuous fashion, waiting if no events are available, or with a timeout that allows control to be returned to the body of the loop on a predictable schedule for heartbeats and other activities.

For reading events from all partitions in a production scenario, we strongly recommend using the EventProcessorClient or EventProcessor<TPartition> over the EventHubConsumerClient.

Event Processor Client

The EventProcessorClient is intended to serve as the primary consumer of events in production scenarios for the majority of workloads. It is responsible for reading and processing events for all partitions of an Event Hub and collaborates with other EventProcessorClient instances using the same Event Hub and consumer group to balance work between them. A high degree of fault tolerance is built-in, allowing the processor to be resilient in the face of errors.

A key feature of the EventProcessorClient is allowing the application to track which events have been processed and persisting them to some form or durable storage so that progress is resilient across failures and observed as instances scale up or down and load-balance. This process is commonly referred to as checkpointing and the persisted state as a checkpoint. This version of the EventProcessorClient supports only Azure Storage Blobs as a backing store; the EventProcessor supports implementing other storage providers.

The EventProcessorClient is located in the Azure.Messaging.EventHubs.Processor package.

Specialized

Partition Receiver

The PartitionReceiver is responsible for consuming events from a specific partition of an Event Hub, offering a lower-level API with greater control over resource use. The PartitionReceiver utilizes a batch-based pull model for applications requesting events. This allows Event Hubs service communication and network activities to be deterministic, initiated by an application request and avoids the need for implicit background operations.

Each PartitionReceiver represents a single consumer scope with one lifespan, able to read events from a partition in a stateful manner and with predictable resource use. Because each instance is self-contained, the PartitionReceiver supports creating multiple receivers for the same partition concurrently, with each able to independently read from a different location in the event stream.

Event Processor

The EventProcessor provides a base to create a custom client for reading and processing events for all partitions of an Event Hub. It collaborates to balance work with other clients derived from EventProcessor<TPartition> as well as EventProcessorClient instances using the same Event Hub and consumer group. A high degree of fault tolerance is built-in, allowing the derived processors to be resilient in the face of errors.

Though it serves a similar role to the EventProcessorClient, the EventProcessor<TPartition> provides a lower-level API with less ceremony and greater control over resource use. It offers native batch-based processing via method override rather than using the event handler pattern to process a single event at a time. The EventProcessor<TPartition> also enables defining a custom partition context for applications that wish to track additional state while processing.

While the EventProcessorClient is an opinionated implementation strongly bound to Azure Storage Blobs, the EventProcessor<TPartition> has no affiliation to a specific storage provider, allowing data-bound operations to be fully controlled by the derived client via method overload.

Azure Event Hubs Information

The Event Hubs client library offers an extensive set of samples, covering many of the core concepts of messaging and Event Hubs as well as usage of the client. They are organized by package and can be found in the GitHub repository:

More detail on the design and philosophy for the PartitionReceiver and EventProcessor<TPartition> can be found in their respective design documents:

General information on Azure Event Hubs can be found on the Microsoft Docs site under the product documentation. Some key documents are:

Azure SDK Releases

Azure SDK Blog Contributions

Thank you for reading this Azure SDK blog post! We hope that you learned something new and welcome you to share this post. We are open to Azure SDK blog contributions. Please contact us at azsdkblog@microsoft.com with your topic and we’ll get you set up as a guest blogger.

Azure SDK Links

Category
Azure SDK
Topics
.NETazure-sdkEvent Hubs

Author

Jesse Squire
Jesse Squire
Senior Software Engineer

My love for technology began at age 6 when I wrote my first lines of BASIC code on a TI-99/4A. From the moment that I saw my name printed across the screen, I knew what I wanted to be when I grew up. More than twenty years of professional experience later, I still hold onto that child-like sense of wonder and feel incredibly fortunate that I’ve been able to build a career around something which I’m passionate about and would be doing as a hobby regardless. I am an engineer on the Azure ...

More about author

0 comments

Discussion are closed.

Read next