promptflow-serve<\/code> and our recommendations.<\/p>\nTest Scenario<\/h2>\n
Before testing the entire application, we created a sample flow that mimicked part<\/strong> of our real flow. It contained a fan-out and fan-in flow to replicate LLM nodes like guardrails running in parallelly and a final node that does an API call to an external dependency. This API call was made to a mock HTTP API<\/strong>. A synthetic delay was added to each of the parallel nodes to replicate LLM calls.<\/p>\nThe harness contained these components:<\/p>\n
\n- A mock HTTP API that acts as a service used by your PromptFlow flow.<\/li>\n
- The following PromptFlow flows that are hosted using
pf-serve<\/code>.\n\n- Synchronous flow hosted using WSGI<\/a>.<\/li>\n
- Asynchronous flow hosted using ASGI<\/a> and async Python functions<\/a> as PromptFlow nodes.<\/li>\n<\/ol>\n<\/li>\n
- Locust<\/a> load test generator.<\/li>\n
- Makefile, scripts and docker-compose file to orchestrate the tests.<\/li>\n<\/ul>\n
The test harness and example used to create the test scenarios discussed has been contributed to the PromptFlow repository via a pull request here<\/a>.<\/p>\nFlow<\/h3>\n
The directed acyclic graph (DAG) for both the flows are shown below.<\/p>\n
![\"flow\"](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2024\/09\/flow.png\")
<\/p>\n
\n- Parallel
nodes 1, 2 and 3<\/code> have a synthetic delay of 0.25ms<\/code> to simulate an LLM call like guardrails.<\/li>\n- The
chat<\/code> node makes a HTTP call to the mock API to simulate a downstream service call.<\/li>\n<\/ul>\nHost Runner Options, Synchronous and Asynchronous Nodes<\/h3>\n
The aim of our selected variations were to test how pf-serve<\/code> behaves when using the default WSGI<\/a> runner (Gunicorn<\/a>) compared to an ASGI<\/a> runner (FastApi<\/a>) with the combination of async PromptFlow nodes.<\/p>\nThe load was generated using Locust<\/a> and had a maximum of 1000 concurrent users<\/strong> with a hatch rate of 10<\/strong>. The test was run for 5 minutes<\/strong>. We ran each combination with 8 workers and 8 threads per worker<\/strong>. The test was run on WSL which had 16GB of memory and 8 logical processors<\/strong>. The guidance around concurrency can be found here<\/a>.<\/p>\nIn hindsight, 1000<\/code> concurrent users were exhausting the limited resources available on the host environment. We not only had the test harness but also the mock API running in the same host. Ideally the mock API should have been run elsewhere so it would not interfere with the test harness. But our aim was to find patterns and bottlenecks rather than be super precise around maximum achievable throughput.<\/strong><\/p>\nIf you want to find a more accurate number for the sustainable concurrent users for your environment, run the included load test against the mock API endpoint provided in the benchmark suite. This will give you sense of what ASGI (FastApi<\/code>) can support in your environment without PromptFlow in the mix. You can also use this result as a guide to compare the throughput to the sync and async variants (where PromptFlow is in the mix).<\/p>\nSync vs Async Nodes<\/h3>\n
The PromptFlow @tool<\/code> annotation supports both sync and async functions. Hence both of the below code examples are valid but as you will see later, they have a massive performance impact as the sync example blocks the thread.<\/p>\nThe sync example uses the requests<\/a> library to make a synchronous call to the mock API.<\/p>\nimport os\r\nimport time\r\n\r\nimport requests\r\nfrom promptflow.core import tool\r\n\r\n@tool\r\ndef my_python_tool(node1: str, node2: str, node3: str) -> str:\r\n\r\n start_time = time.time()\r\n\r\n # make a call to the mock endpoint\r\n url = os.getenv(\"MOCK_API_ENDPOINT\", None)\r\n if url is None:\r\n raise RuntimeError(\"Failed to read MOCK_API_ENDPOINT env var.\")\r\n\r\n # respond with the service call and tool total times\r\n response = requests.get(url)\r\n if response.status_code == 200:\r\n response_dict = response.json()\r\n end_time = time.time()\r\n response_dict[\"pf_node_time_sec\"] = end_time - start_time\r\n response_dict[\"type\"] = \"pf_dag_sync\"\r\n return response_dict\r\n else:\r\n raise RuntimeError(f\"Failed call to {url}: {response.status_code}\")<\/code><\/pre>\nThe below async example uses aiohttp<\/a> to make an async call to the mock API which allows the node function to be async as well.<\/p>\nimport os\r\nimport time\r\n\r\nimport aiohttp\r\nfrom promptflow.core import tool\r\n\r\n@tool\r\nasync def my_python_tool(node1: str, node2: str, node3: str) -> str:\r\n\r\n start_time = time.time()\r\n\r\n # make a call to the mock endpoint\r\n url = os.getenv(\"MOCK_API_ENDPOINT\", None)\r\n if url is None:\r\n raise RuntimeError(\"Failed to read MOCK_API_ENDPOINT env var.\")\r\n\r\n async with aiohttp.ClientSession() as session:\r\n async with session.get(url) as response:\r\n if response.status == 200:\r\n response_dict = await response.json()\r\n end_time = time.time()\r\n response_dict[\"pf_node_time_sec\"] = end_time - start_time\r\n response_dict[\"type\"] = \"pf_dag_async\"\r\n return response_dict\r\n else:\r\n raise RuntimeError(f\"Failed call to {url}: {response.status}\")<\/code><\/pre>\nCombinations Tested<\/h3>\n\n- WSGI (
gunicorn<\/code>) + Sync PF Nodes<\/li>\n- ASGI (
fastapi<\/code>) + Async PF Nodes<\/li>\n<\/ul>\nIt’s important to note that PromptFlow Docker image based deployment<\/a> uses the WSGI runner (Flask<\/code>) by default. You must opt-in to use FastApi<\/a>.<\/p>\nTest Results<\/h2>\n
\n\n\nMetric<\/strong><\/th>\nWSGI + Sync Nodes<\/strong><\/th>\nASGI + Async Nodes<\/strong><\/th>\n<\/tr>\n<\/thead>\n\n\nRequest Count<\/strong><\/td>\n12,157<\/td>\n 65,554<\/td>\n<\/tr>\n \nFailure Count<\/strong><\/td>\n1<\/td>\n 43<\/td>\n<\/tr>\n \nMedian Response Time (ms)<\/strong><\/td>\n9,900<\/td>\n 1,400<\/td>\n<\/tr>\n \nAverage Response Time (ms)<\/strong><\/td>\n13,779.82<\/td>\n 1,546.90<\/td>\n<\/tr>\n \nMin Response Time (ms)<\/strong><\/td>\n1.07<\/td>\n 0.73<\/td>\n<\/tr>\n \nMax Response Time (ms)<\/strong><\/td>\n48,101.42<\/td>\n 4,212.50<\/td>\n<\/tr>\n \nRequests\/s<\/strong><\/td>\n40.60<\/td>\n 218.85<\/td>\n<\/tr>\n \nFailures\/s<\/strong><\/td>\n0.0033<\/td>\n 0.1435<\/td>\n<\/tr>\n \n50% Response Time (ms)<\/strong><\/td>\n9,900<\/td>\n 1,400<\/td>\n<\/tr>\n \n66% Response Time (ms)<\/strong><\/td>\n17,000<\/td>\n 1,500<\/td>\n<\/tr>\n \n75% Response Time (ms)<\/strong><\/td>\n22,000<\/td>\n 1,700<\/td>\n<\/tr>\n \n80% Response Time (ms)<\/strong><\/td>\n24,000<\/td>\n 1,800<\/td>\n<\/tr>\n \n90% Response Time (ms)<\/strong><\/td>\n34,000<\/td>\n 2,100<\/td>\n<\/tr>\n \n95% Response Time (ms)<\/strong><\/td>\n40,000<\/td>\n 2,400<\/td>\n<\/tr>\n \n98% Response Time (ms)<\/strong><\/td>\n47,000<\/td>\n 2,800<\/td>\n<\/tr>\n \n99% Response Time (ms)<\/strong><\/td>\n48,000<\/td>\n 3,000<\/td>\n<\/tr>\n \n99.9% Response Time (ms)<\/strong><\/td>\n48,000<\/td>\n 4,000<\/td>\n<\/tr>\n \n99.99% Response Time (ms)<\/strong><\/td>\n48,000<\/td>\n 4,100<\/td>\n<\/tr>\n \n100% Response Time (ms)<\/strong><\/td>\n48,000<\/td>\n 4,200<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nThroughput Graph<\/h3>\nWSGI + Sync Nodes<\/h4>\n
![\"sync](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2024\/09\/sync_result_graph.png\")
<\/p>\n
ASGI + Async Nodes<\/h4>\n
![\"async](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2024\/09\/async_result_graph.png\")
<\/p>\n
Detailed Comparison<\/h3>\nRequest Count and Throughput<\/h4>\n\n- Request Count:<\/strong> Async variant handled approximately 5.4 times more requests than Sync setup.<\/li>\n
- Requests\/s:<\/strong> Async variant achieved 218.85 requests\/s compared to Sync setup’s 40.60 requests\/s. This indicates async variant handled requests more efficiently, possibly due to its asynchronous nature.<\/li>\n<\/ul>\n
Response Time<\/h4>\n\n- Median Response Time:<\/strong> Async variant’s median response time (1,400 ms) was significantly lower than Sync setup’s (9,900 ms). This shows that most requests in async variant were completed faster.<\/li>\n
- Average Response Time:<\/strong> Async variant’s average response time (1,546.90 ms) was also significantly lower than Sync setup’s (13,779.82 ms). This highlights async variant’s overall better performance in handling requests.<\/li>\n
- Min Response Time:<\/strong> Async variant’s minimum response time (0.73 ms) was slightly lower than Sync setup’s (1.07 ms), indicating faster handling of the quickest requests.<\/li>\n
- Max Response Time:<\/strong> Async variant’s maximum response time (4,212.50 ms) was much lower than Sync setup’s (48,101.42 ms), suggesting async variant managed peak loads more effectively.<\/li>\n<\/ul>\n
Failure Count and Rate<\/h4>\n\n- Failure Count:<\/strong> Sync setup had only 1 failure, whereas async variant had 43 failures. Despite this, async variant’s higher request count still resulted in a very low failure rate.<\/li>\n
- Failures\/s:<\/strong> Sync setup’s failure rate (0.0033 failures\/s) was lower than async variant’s (0.1435 failures\/s), although this needs to be considered in the context of the much higher load handled by async variant.<\/li>\n<\/ul>\n
Percentiles<\/h4>\n\n- Sync setup:<\/strong> The higher percentiles (75%, 90%, 95%, etc.) showed very high response times, peaking at 48,000 ms. This indicates Sync setup struggled significantly with higher loads, leading to large delays.<\/li>\n
- Async variant:<\/strong> The percentiles for async variant were much lower, with the 99.9th percentile at 4,000 ms. This shows async variant provided more consistent performance under load.<\/li>\n<\/ul>\n
Analysis of Performance Differences<\/h2>\n
The async variant demonstrated significantly better performance compared to the Sync setup application across all key metrics, handling more requests with lower response times and higher throughput. The primary reason for this difference is async variant’s end-to-end asynchronous nature, which allows it to handle I\/O-bound operations more efficiently and manage multiple requests concurrently, unlike Sync setup’s synchronous handling. While async variant had a slightly higher failure rate, this was relatively minor considering the much higher load it managed to process effectively.<\/p>\n
Evidence Of Backpressure In The Sync Setup<\/h3>\n
Backpressure occurs when a system becomes overwhelmed by the volume of incoming requests and cannot process them quickly enough, leading to increased response times and potential failures. Here are the indicators suggesting backpressure in the sync variant:<\/p>\n
\n- High Median and Average Response Times<\/strong>:\n
\n- Median Response Time: 9,900 ms<\/li>\n
- Average Response Time: 13,779.82 ms<\/li>\n<\/ul>\n
These high response times indicate that the sync variant is taking a long time to process requests, which is a sign that it is struggling to keep up with the incoming load.<\/li>\n
- Wide Range of Response Times<\/strong>:\n
\n- Min Response Time: 1.07 ms<\/li>\n
- Max Response Time: 48,101.42 ms<\/li>\n<\/ul>\n
The vast difference between the minimum and maximum response times shows that while some requests are processed quickly, others take an excessively long time, suggesting the system is experiencing periods of high load that it cannot handle efficiently.<\/li>\n
- High Percentile Response Times<\/strong>:\n
\n- 75% Response Time: 22,000 ms<\/li>\n
- 90% Response Time: 34,000 ms<\/li>\n
- 95% Response Time: 40,000 ms<\/li>\n
- 99% Response Time: 48,000 ms<\/li>\n
- 99.9% Response Time: 48,000 ms<\/li>\n<\/ul>\n
The high response times at these percentiles indicate that a significant proportion of requests are delayed, further suggesting the application is overwhelmed.<\/li>\n
- Max Response Time<\/strong>:\n
\n- The maximum response time of 48,101.42 ms is extremely high and indicates that under peak load, some requests are waiting an excessively long time to be processed.<\/li>\n<\/ul>\n<\/li>\n
- Low Requests per Second (Requests\/s)<\/strong>:\n
\n- WSGI sync setup: 40.60 Requests\/s<\/li>\n
- ASGI async setup: 218.85 Requests\/s<\/li>\n<\/ul>\n
The sync variant handled far fewer requests per second compared to the async variant, indicating it is less capable of handling high loads efficiently.<\/li>\n<\/ol>\n
These metrics collectively suggest that the sync variant is experiencing backpressure<\/strong>. The high and variable response times, coupled with the lower throughput, indicate that the application cannot process requests quickly enough under high load, resulting in delays and potentially dropped requests. In contrast, the async variant demonstrates significantly better performance and is more resilient to high loads<\/strong>, thanks to its asynchronous processing model.<\/p>\nThe sync setup blocks the thread while waiting for the nodes to finish executing and thread pool exhaustion occurs when all available threads in the thread pool are occupied<\/strong> and new requests cannot be processed until some threads are freed up. This situation can lead to backpressure, where incoming requests are delayed or queued because the application server cannot handle them immediately. Increasing the worker and thread count may help but ultimately still suffers from thread blocking operations.<\/p>\nWhile async setup also shows signs of backpressure, such as higher response times for a small percentage of requests and a non-zero failure rate, it performs significantly better than sync setup under similar conditions. The asynchronous nature allows it to handle higher loads more efficiently, but it is still not entirely immune to the effects of backpressure when pushed to its limits. The signs of backpressure in FastAPI<\/code> are much less severe compared to that of the synchronous gunicorn<\/code> setup, highlighting its superior performance in handling concurrent requests.<\/p>\ntldr<\/h2>\nThe experiment showed that the bottleneck on the sync variant was the PromptFlow application itself, where as with the async variant the limiting factor was the system resources. This is an important learning showing that the async option achieves more with the same resources<\/strong>. This may seem obvious but there is responsibility on your part as a developer to ensure that the Python functions are async compatible (using the async await pattern and picking the right libraries for I\/O) so the PromptFlow flow executor and ASGI hosting can take advantage of it.<\/p><\/blockquote>\nBonus Reading: Why Are There Relatively High Network Failures In The Async Variant?<\/h2>\n
During the test, the async setup resulted in 44 network errors while the sync setup only had 1.<\/p>\n
The observed errors were:<\/p>\n
\n\n\nError<\/th>\n Description<\/th>\n<\/tr>\n<\/thead>\n \n\nRemoteDisconnected('Remote end closed connection without response')<\/code><\/td>\nThis error occurs when the client closes the connection abruptly. In the context of a high-load environment, this can happen if the server is too slow to respond, causing the client to time out and close the connection.<\/td>\n<\/tr>\n \nConnectionResetError(104, 'Connection reset by peer')<\/code><\/td>\nThis error indicates that the server closed the connection without sending a response. This can happen if the server is overwhelmed and cannot handle new incoming connections or if it runs out of resources to maintain open connections.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nThese can be indicative of resource limits on the host.<\/p>\n
\n- CPU Limits: If the server’s CPU is fully utilized, it might not be able to process incoming requests in a timely manner, leading to clients timing out and closing connections.<\/li>\n
- Memory Limits: If the server runs out of memory, it might kill processes or fail to accept new connections, leading to connection resets.<\/li>\n<\/ul>\n
Remember that we ran the mock API on the same environment and it required CPU, memory and network resources as well. As mentioned earlier the mock API is competing for resources with the test harness in this shared environment.<\/p>\n
Explaining The Abrupt Changes In Requests Per Second and Response Time<\/h3>\n
If you looked at the time series graph closely, you would have noticed that there is some abrupt changes in the throughput and response times.<\/p>\n
![\"network](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2024\/09\/async_exceptions.png\")
<\/p>\n
We observed that these occurred at the same timestamps when the above mentioned network errors occurred. Further indicating that this happened due to resource limitations.<\/p>\n
Summary Of Findings And Recommendations<\/h2>\n
As observed, the async setup demonstrated significantly higher throughput and better response times. The only metric that was worse in the async setup was the number of network exceptions occurred but that was most likely due to the limitation of the memory and CPU in the environment the test was run in.<\/p>\n
\n- Utilise an async supported http client<\/a> like
aiohttp<\/code> or httpx<\/code> when calling downstream APIs or LLM endpoints. This would allow you to bubble up the async await pattern up to the node function level and allow PromptFlow flow executor to take advantage of it.<\/li>\n- Start using FastAPI as the runner<\/a> for
pf-serve<\/code> by setting the PROMPTFLOW_SERVING_ENGINE=fastapi<\/code> environment variable.<\/li>\n<\/ul>\nThese recommendations are easy to implement and should be good defaults for most scenarios. We tested the new flex flow from PromptFlow with an async setup and it behaved similar to the static DAG based flow.<\/p>\n
Closing<\/h2>\n