{"id":14644,"date":"2023-05-04T00:00:00","date_gmt":"2023-05-04T07:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cse\/?p=14644"},"modified":"2024-07-18T11:51:21","modified_gmt":"2024-07-18T18:51:21","slug":"guide-to-workflow-engines","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/guide-to-workflow-engines\/","title":{"rendered":"Hitchhikers Guide to Workflow Engines"},"content":{"rendered":"<blockquote>\n<p>Don&#8217;t Panic! This guide is designed to help you understand the basics of workflow engines, with no prior knowledge needed!<\/p>\n<\/blockquote>\n<p>We&#8217;ll introduce workflow engines and their qualities and features, and provide some insights into when and why you might want to\nuse one. Moreover, we&#8217;ll highlight key considerations to keep in mind when selecting an engine that suits your specific needs.\nThis will equip you with a good starting point for making an informed decision.<\/p>\n<h2>Workflow Engines: Orchestrating Complex Processes with Precision and Scalability<\/h2>\n<p>One of our large customers in the automotive industry approached us with the perfect scenario for a workflow engine. They wanted\nto read data from their PLCs (Programmable Logic Controllers) and programmatically change values based on third-party systems,\nvalues read from the PLC, or provided parameters. They also wanted the process to be developed by their production engineers.<\/p>\n<p>This scenario demonstrates the need for an effective way to manage and automate complex processes, which is where workflow\nengines come into play. They serve as a powerful foundation for building software applications that can tackle complex\nchallenges and adapt to ever-changing requirements. Workflow engines empower developers to create solutions that dynamically\nscale to meet demand, reuse components for efficiency, provide deep insights into system health, and recover gracefully from\nerrors. They offer a robust framework to define, execute, and monitor workflows across various systems and applications.<\/p>\n<p>Now that we have the perfect scenario for using a workflow engine, it&#8217;s up to us to decide which one best suits our needs and\nrequirements, allowing us to fully harness the power of customizable and scalable processes.<\/p>\n<h2>Key Considerations for Selecting a Workflow Engine: Qualities and Features<\/h2>\n<p>Workflow engines possess a range of characteristics that enable them to execute processes efficiently. These features will be\ndiscussed in greater detail in the paragraphs that follow. Before diving in, it is crucial to establish a common understanding\nof certain terms. In the next section, a process conducted by a workflow engine will be referred to as an &#8220;activity&#8221; (also\ncalled Task, Step, or Job) which can be in a <code>successful<\/code> <img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2023\/05\/workflow-engine-check-v2.png\" alt=\"Successful Activity\" \/>, <code>failed<\/code>\n<img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2023\/05\/workflow-engine-failed-v2.png\" alt=\"Failed Activity\" \/> or <code>not started<\/code> <img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2023\/05\/workflow-engine-not-started-v2.png\" alt=\"Not Started activity\" \/> state. A workflow\ndefinition is the process of chaining\nactivities together to form a sequence and a workflow execution is the current state of a system running a specified definition.<\/p>\n<p>To help you understand the different uses of an engine, we&#8217;ve put together a list of important qualities and features that\nengines should have, which we&#8217;ll discuss in the following paragraphs.<\/p>\n<blockquote>\n<p>The features outlined below provide a general idea of what you can expect from a workflow engine. However, it&#8217;s important to\ndrill deeper into the specifics, as each engine may have various implementations that are either desirable or undesirable\ndepending on your needs.<\/p>\n<\/blockquote>\n<h3>Scalability<\/h3>\n<p>Scalability is a crucial feature for any application, and workflow engines enable your code to handle an increasing number of\nexecutions on a large scale. For instance, a workflow engine can efficiently manage and coordinate numerous PLCs across a\nfactory floor, even as the number of devices and their complexity grows.<\/p>\n<p>A scalable workflow engine should offer these features:<\/p>\n<ul>\n<li><strong>Multi-cluster replication:<\/strong> Distribute and synchronize workflow data across multiple clusters for high availability and\nfault tolerance.<\/li>\n<li><strong>Archival of old data:<\/strong> Store and manage historical workflow data for compliance, auditing, and analysis purposes.<\/li>\n<li><strong>Scalable persistence stores:<\/strong> Use scalable storage solutions to accommodate growing workflow data and activity loads.<\/li>\n<\/ul>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2023\/05\/workflow-engine-scalable-v3.png\" alt=\"Multiple workflow executions shown in parallel\" \/><\/p>\n<h3>Reusability<\/h3>\n<p>Workflow engines should enable developers to create reusable activities and components that can be easily integrated into new\nworkflows. This not only saves time but also ensures consistent behavior across different processes. For instance, a reusable\npayment processing activity can be used in various workflows, such as\nonline purchases, subscription renewals, or refunds.<\/p>\n<p>A workflow engine that enables reusability should offer these features:<\/p>\n<ul>\n<li><strong>Sub-workflow execution:<\/strong> Allow workflows to call and execute other workflows.<\/li>\n<li><strong>Activity reuse:<\/strong> Enable activities or components to be reused across multiple workflows.<\/li>\n<li><strong>Shared libraries or templates:<\/strong> Provide a repository of predefined workflow patterns, activities, or templates to simplify\nand standardize workflow creation.<\/li>\n<\/ul>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2023\/05\/workflow-engine-reusable-v3.png\" alt=\"Use of a sub-workflow shown within another workflow\" \/><\/p>\n<h3>Observability<\/h3>\n<p>Monitoring and maintaining the health of a system is an essential to keep processes running smoothly. Workflow engines should\noffer the capability to query and observe the state of every activity, workflow, group of workflows, or the entire system. For\nexample, a system\nadministrator can easily track the progress of a long-running data processing workflow, identifying bottlenecks or errors in\nreal-time.<\/p>\n<p>A workflow engine with robust observability should provide these features:<\/p>\n<ul>\n<li><strong>Metrics<\/strong>: Collect and analyze performance indicators for workflows, activities, and resources to optimize and monitor the\nsystem.<\/li>\n<li><strong>Tracing<\/strong>: Leverage distributed tracing to visualize the call graph of a workflow, including its activities and any child\nworkflows. This allows you to debug and diagnose issues more effectively<\/li>\n<li><strong>Logging<\/strong>: Record and store logs for workflow execution, errors, and events to aid in troubleshooting and auditing.<\/li>\n<li><strong>Alerting<\/strong>: Send alerts or trigger actions based on predefined conditions, such as workflow failures, high resource usage,\nor long-running activities.<\/li>\n<\/ul>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2023\/05\/workflow-engine-observable-v3.png\" alt=\"Workflow that changes behavior based on failing activity\" \/><\/p>\n<h3>Resilience<\/h3>\n<p>Resilience is a significant advantage of workflow engines, as they provide the ability to retry activities and workflows\nwith different policies. This feature helps ensure that transient errors, such as temporary network outages, do not cause the\nentire process to fail.<\/p>\n<p>A resilient workflow engine should provide the following features:<\/p>\n<ul>\n<li><strong>Retry policies for activities and workflows<\/strong>: Automatically retry failed activities or workflows based on user-defined\npolicies.<\/li>\n<li><strong>Timeouts for activities and workflows<\/strong>: Set limits on how long activities or workflows can run or wait for action,\npreventing stalled processes.<\/li>\n<li><strong>Fallback strategies<\/strong>: Define alternative actions or notifications when retries fail or certain conditions are met.<\/li>\n<\/ul>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2023\/05\/workflow-engine-resilient-v3.png\" alt=\"Workflow that shows transitioning from a failing into a successful activity\" \/><\/p>\n<h3>Durability<\/h3>\n<p>Durability ensures that workflows can process highly complex workflows, run indefinitely, or wait for action for hours or even\ndays without losing data or state. For example, an approval process in a large organization might require multiple levels of\nsign-offs, and the associated workflow must be able to wait for the necessary approvals, even if it takes several days. With\ndurable workflow engines, developers can create long-running processes that maintain their state and data integrity across\nextended periods, ensuring that no critical information is lost during the execution.<\/p>\n<p>A durable workflow engine should offer these features:<\/p>\n<ul>\n<li><strong>Timers<\/strong>: Schedule workflows to begin execution at predetermined times or intervals.<\/li>\n<li><strong>Sleep functionality<\/strong>: Allow workflows to pause or wait for action without consuming excessive resources, supporting\nlong-running processes.<\/li>\n<li><strong>Visibility into the current and past state<\/strong>: Provide insights into the current status and historical execution of\nworkflows, enabling better tracking, management, and auditing.<\/li>\n<\/ul>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2023\/05\/workflow-engine-durable-v3.png\" alt=\"A long running workflow represented by 3 dots inside of an activity bubble\" \/><\/p>\n<h2>Comparing Features of Workflow Engines: Code-based vs. DSL-based<\/h2>\n<p>Workflow engines have traditionally utilized domain-specific languages (DSLs) such as JSON or YAML to represent workflows. This\napproach offers several advantages, including ease of deserialization and serialization, which enables the creation,\ntransmission, and validation of workflows using well-defined schemas e.g. JSON, YAML.<\/p>\n<p>However, as workflows grow in size and complexity, developers may require more advanced features. Recently, workflow engines\nlike DTFx and Temporalio have emerged, allowing developers to use code for designing workflows.<\/p>\n<p>In this chapter, we compare the features of both code-based and DSL-based workflow engines to provide a balanced perspective.<\/p>\n<h3>Type of workflow definitions ( Workflow as Code \/ Workflow as DSL )<\/h3>\n<p>There are two primary categories of workflow definitions, each catering to a distinct audience. One focuses on developers, while the\nother targets business users. Both share common features, but also possess unique elements specifically designed for their\nrespective audiences.<\/p>\n<h4>Workflow as DSL<\/h4>\n<p>Domain-Specific Language, or DSL, refers to a usually declarative language specifically built to address\nchallenges within a particular domain or industry.<\/p>\n<p>DSLs enable domain experts to define and configure workflows without need for extensive programming skills or understanding of\nthe underlying workflow engine. By streamlining the process of\ncreating and modifying workflows, DSLs not only enhance their readability but also increase their maintainability.<\/p>\n<p>You can see a very simple example here which describes a workflow with just one activity that writes to console.<\/p>\n<pre><code class=\"language-yaml\">Id: HelloWorld\r\nActivities:\r\n- Id: Hello\r\n  ActivityType: Console.Write.HelloWorld, MyApp<\/code><\/pre>\n<h4>Workflow as Code<\/h4>\n<p>Workflow-as-code definitions are typically comprised of three essential components. First, there are activities, which serve as\nthe building blocks for executing tasks within the workflow. Second, a workflow definition is created, written in code, to\norganize these activities. Finally, a wrapper is utilized around these activities, enabling the chosen framework to manage input\nand output. This allows for seamless internal transfer between executing instances and promotes scalability.<\/p>\n<p>In this pseudo example you can see how a workflow is created directly in the code with <code>function void workflow(Context context)<\/code>\nand then the function <code>function string activity()<\/code> is utilized inside of <code>context.RunActivity<\/code>.<\/p>\n<pre><code class=\"language-csharp\">async function string activity() {\r\n  return \"Hello World\";\r\n}\r\n\r\nasync function void workflow(Context context) {\r\n   string text = await context.RunActivity(activity)\r\n}<\/code><\/pre>\n<h4>Comparing the two types of workflow engines<\/h4>\n<p>Advantages of DSL-based Workflow Engines<\/p>\n<ul>\n<li><strong>Visual Editing<\/strong>: These engines often provide a user interface where the workflow can be visually edited and viewed.<\/li>\n<li><strong>Integrations<\/strong>: Many DSL-based engines support pre-built integrations with various external systems and services, such as\ndatabases, messaging platforms, or third-party APIs. These integrations can be beneficial, as they simplify the process of\nconnecting and interacting with these external components, saving time and effort compared to developing custom integrations\nfrom scratch.<\/li>\n<li><strong>Dynamic Configuration<\/strong>: As DSL-based workflows are essentially configuration files, they can be easily stored in blob\nstorages and loaded into engines without deployment, offering greater flexibility and ease of use when updating or adding new\nworkflows.<\/li>\n<li><strong>Accessible<\/strong>: DSL-based engines are more accessible to non-developers, thanks to the use of simplified languages geared\ntowards non-technical users.<\/li>\n<li><strong>Enhancement Capabilities<\/strong>: The DSL-based approach allows for the augmentation of code with additional configurations\nretrieved from other systems that may be similar in nature, making it easier to adapt and extend the workflow\nfunctionality for non-technical users.<\/li>\n<\/ul>\n<p>Advantages of Code-based Workflow Engines<\/p>\n<ul>\n<li><strong>Type Safety<\/strong>: Activities often require access to prior data or a global state in order to process input data. Typically,\nthis data is untyped, which can result in overlooked edge cases during workflow development.<\/li>\n<li><strong>IDE Support<\/strong>: When building workflows, the DSL method falls short in providing IDE Support for recommending activities, as\nwell as input and output types.<\/li>\n<li><strong>Debugger<\/strong>: Generally, DSL-based engines lack debugger support for quickly identifying code bugs, instead relying on error\nreporting and logging.<\/li>\n<li><strong>Testability<\/strong>: Code-based engines allow developers to write unit tests for individual workflow components, as well as\nintegration tests for the entire workflow. This ensures that workflows function as expected, improves code quality, and reduces\nthe likelihood of introducing errors or inconsistencies during development.<\/li>\n<\/ul>\n<p>Taking into account the advantages of both code-based and DSL-based workflow engines allows organizations to make well-rounded\ndecisions when selecting an engine that aligns with their unique needs.<\/p>\n<h2>Criteria for Evaluating Workflow Engines<\/h2>\n<p>To effectively choose a workflow engine for your specific use case, it is essential to evaluate the engines based on a set of\ncriteria. The primary decision you must make is determining the target audience, either developers or non-technical users, as this will\nsignificantly influence the suitability of the workflow engine for your project requirements. This chapter outlines the key\nevaluation criteria to consider when selecting a workflow engine for each audience.<\/p>\n<blockquote>\n<p>Tip: Before diving into engine comparisons, clearly define your target audience, be it developers or non-technical users. This\nstep will help streamline your selection process by focusing on the features and capabilities most relevant to your audience.<\/p>\n<\/blockquote>\n<p>As previously stated, the two primary types of workflow engines cater to distinct audiences.<\/p>\n<ul>\n<li><strong>Developers (Workflow as Code)<\/strong>: Workflow engines geared towards developers emphasize code-based configuration,\nextensibility, and integration with existing developer tools and platforms.<\/li>\n<li><strong>End users (Workflow as DSL)<\/strong>: Workflow engines designed for non-technical users prioritize user-friendly interfaces, visual design\ncapabilities, and a gentle learning curve.<\/li>\n<\/ul>\n<p>When evaluating workflow engines, consider the following criteria:<\/p>\n<ul>\n<li><strong>Performance<\/strong>: Assess the engine&#8217;s ability to meet the specific performance requirements of your use case.<\/li>\n<li><strong>Integrations<\/strong>: Evaluate the built-in integrations offered by various engines, particularly those targeting non-technical users, to\nsimplify your use case implementation.<\/li>\n<li><strong>Popularity<\/strong>: Consider the community support and usage for an engine, as evidenced by GitHub Stars and pulls on the platform\nwhere it is hosted.<\/li>\n<li><strong>Features<\/strong>: Determine which <a href=\"#key-considerations-for-selecting-a-workflow-engine-qualities-and-features\">features<\/a> are\nessential for your use case and evaluate engines based on their implementation of\nthose features.<\/li>\n<li><strong>Scalability<\/strong>: Consider the scalability requirements of your use case and whether the engine is designed to accommodate such\ndemands.<\/li>\n<li><strong>Licensing<\/strong>: Consider the licensing requirement of your use case and see if workflow engine license fits into your project\nuse case.<\/li>\n<li><strong>Programming Language<\/strong>: A key criteria for selecting the engine is utilizing an engine where building custom workflows for\nyou and your team is easy and utilizing the right libraries is simple.<\/li>\n<li><strong>Integration Type<\/strong>: Evaluate the desired type of the workflow engine, such as a library, framework, API, or fully-fledged\napplication, based on your project&#8217;s specific needs and how it should integrate with your existing systems or development\nprocesses.<\/li>\n<\/ul>\n<p>By using these evaluation criteria, you can better understand the strengths and weaknesses of different workflow engines,\nenabling you to choose the most suitable one for your specific requirements and target audience.<\/p>\n<h2>Reference table for different engines<\/h2>\n<p>This is a short list of popular engines with active community support, for a more exhaustive curated list have a look at\n<a href=\"https:\/\/github.com\/meirwah\/awesome-workflow-engines\">awesome-workflow-engines<\/a>.<\/p>\n<table>\n<thead>\n<tr>\n<th>Name<\/th>\n<th>Programming Language<\/th>\n<th>Type<\/th>\n<th>Links<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Apache Airflow<\/td>\n<td>Python<\/td>\n<td>Code<\/td>\n<td><a href=\"https:\/\/airflow.apache.org\/\">Website<\/a> , <a href=\"https:\/\/github.com\/apache\/airflow\/\">GitHub<\/a><\/td>\n<\/tr>\n<tr>\n<td>n8n<\/td>\n<td>JavaScript\/TypeScript<\/td>\n<td>DSL<\/td>\n<td><a href=\"https:\/\/n8n.io\/\">Website<\/a> , <a href=\"https:\/\/github.com\/n8n-io\/n8n\">GitHub<\/a><\/td>\n<\/tr>\n<tr>\n<td>DTFx<\/td>\n<td>C#<\/td>\n<td>Code<\/td>\n<td><a href=\"https:\/\/github.com\/Azure\/durabletask\">GitHub<\/a><\/td>\n<\/tr>\n<tr>\n<td>Temporal<\/td>\n<td>Go, Java, PHP, Python, TypeScript<\/td>\n<td>Code<\/td>\n<td><a href=\"https:\/\/temporal.io\/\">Website<\/a>, <a href=\"https:\/\/github.com\/temporalio\/temporal\">GitHub<\/a><\/td>\n<\/tr>\n<tr>\n<td>Node-RED<\/td>\n<td>JS<\/td>\n<td>DSL<\/td>\n<td><a href=\"https:\/\/nodered.org\/\">Website<\/a>, <a href=\"https:\/\/github.com\/node-red\/node-red\">Github<\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Conclusion<\/h2>\n<p>Throughout this guide, we have explored the essential elements of workflow engines, their qualities and features, and the\ncritical factors to consider when selecting the right engine for your specific use case. By understanding the unique\nrequirements of your project, whether it is geared towards developers or non-technical users, and assessing the scalability, reusability,\nobservability, resilience, and durability of each engine, you can make an informed decision that best meets your needs. With\nthis guide in mind, we were able to choose a workflow engine that fit our customer&#8217;s needs, and we are confident that you will\nbe able to do the same.<\/p>\n<blockquote>\n<p>Remember that the ideal workflow engine should not only address your current requirements but also be\ncapable of adapting to the ever-evolving challenges of your business, ensuring that you can continue to create efficient and\nreliable workflows that drive success.<\/p>\n<\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>This guide explores workflow engines, their essential features, and types, while offering insights and evaluation criteria to help you choose the right solution for your specific use case.<\/p>\n","protected":false},"author":118100,"featured_media":14662,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[3397],"class_list":["post-14644","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cse","tag-workflow-engines"],"acf":[],"blog_post_summary":"<p>This guide explores workflow engines, their essential features, and types, while offering insights and evaluation criteria to help you choose the right solution for your specific use case.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/14644","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/118100"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=14644"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/14644\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/14662"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=14644"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=14644"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=14644"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}