There are many possible approaches to implement retries logic with exponential backoff also depending on the context/protocol, etc. (Database connections, HTTP requests, etc.)
Retries with exponential backoff is a technique that assumes failure by nature and attempts to retry the operation, with an exponentially increasing wait time, until a maximum retry count has been reached. This technique embraces the fact that intermittently, cloud resources may be unavailable more than a few seconds, for any reason out of your control.
An important case is about SQL Azure Databases that may be moved to another server at any time for load balancing reasons, causing the database from being unavailable for a few seconds.
Another possibility could be if you have SQL Server deployed as a container and for some reason (like when spinning up the containers with docker-compose up) the SQL Server container is still not ready for other microservices/containers, like we do at the eShopOnContainers reference application for the dev/test environment.
Resilient Entity Framework Core Sql Connections
In regards the Azure SQL DB case, Entity Framework Core already provides internal database connection resiliency and retry logic, but you need to enable your desired execution strategy per DbContext connection if you want to have resilient EF Core connections.
For instance, the following code at the EF Core connection level is enabling resilient SQL connections that will re-try if it gets any temporal failure like it is possible when using Azure SQL Database or SQL Server deployed as a container if it is not ready.
| //Startup.cs from any ASP.NET Core Web API
public class Startup { //Other code… //… public IServiceProvider ConfigureServices(IServiceCollection services) { //…      services.AddDbContext<OrderingContext>(options => { options.UseSqlServer(Configuration[“ConnectionString”],                                           sqlServerOptionsAction: sqlOptions =>                                           {                                                 sqlOptions.EnableRetryOnFailure(maxRetryCount: 5,                                                 maxRetryDelay: TimeSpan.FromSeconds(30),                                                 errorNumbersToAdd: null);                                           }); }); //… } } |
Execution strategies and explicit transactions using BeginTransaction()
When retries are enabled in EF Core connections, each operation you perform via EF Core becomes its own retriable operation, i.e. each query and each call to SaveChanges() will be retried as a unit if a transient failure occurs.
However, if your code initiates a transaction using BeginTransaction() you are defining your own group of operations that need to be treated as a unit, i.e. everything inside the transaction would need to be played back shall a failure occur. You will receive an exception like the following if you attempt to do this when using an execution strategy.
System.InvalidOperationException: The configured execution strategy ‘SqlServerRetryingExecutionStrategy’ does not support user initiated transactions. Use the execution strategy returned by ‘DbContext.Database.CreateExecutionStrategy()’ to execute all the operations in the transaction as a retriable unit.
The solution, is to manually invoke the execution strategy with a delegate representing everything that needs to be executed. If a transient failure occurs, the execution strategy will invoke the delegate again, as in the following code implemented in eShopOnContainers when using two multiple DbContexts related to the CatalogContext and the IntegrationEventLogContext when updating a product and saving the ProductPriceChanged integration event that needs to use a different DbContext.
| public async Task<IActionResult> UpdateProduct([FromBody]CatalogItem productToUpdate)
{ //Other code… //Update current product catalogItem = productToUpdate; // Use of an EF Core resiliency strategy when using multiple DbContexts // within an explicit BeginTransaction(): // See: https://docs.microsoft.com/en-us/ef/core/miscellaneous/connection-resiliency var strategy = _catalogContext.Database.CreateExecutionStrategy();    await strategy.ExecuteAsync(async () => { // Achieving atomicity between original Catalog database operation and the // IntegrationEventLog thanks to a local transaction using (var transaction = _catalogContext.Database.BeginTransaction()) { _catalogContext.CatalogItems.Update(catalogItem); await _catalogContext.SaveChangesAsync(); //Save to EventLog only if product price changed if (raiseProductPriceChangedEvent) await _integrationEventLogService.SaveEventAsync(priceChangedEvent); transaction.Commit(); } }); //Other code } |
The code above is actual code we’re implementing in the eShopOnContainers reference application, so you can see it in action across a microservice + Docker containers application here:
https://github.com/dotnet/eShopOnContainers   (The code above, currently in the DEV branch, will be merged to the MASTER branch pretty soon).
There are other scenarios with Retries with Exponential Backoff related to protocols like HTTP. I’ll blog about it in upcoming posts. 🙂
0 comments