{"id":7587,"date":"2024-02-22T07:00:51","date_gmt":"2024-02-22T15:00:51","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cosmosdb\/?p=7587"},"modified":"2024-04-30T19:28:28","modified_gmt":"2024-05-01T02:28:28","slug":"mongodb-to-azure-cosmos-db-for-nosql-migration-learnings-from-java-sdk-v4-micro-optimizations","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cosmosdb\/mongodb-to-azure-cosmos-db-for-nosql-migration-learnings-from-java-sdk-v4-micro-optimizations\/","title":{"rendered":"MongoDB to Azure Cosmos DB for NoSQL Migration &#8211; Learnings from Java SDK V4 Micro-Optimizations"},"content":{"rendered":"<p><em><span class=\"TextRun SCXW14107897 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW14107897 BCX8\">The authors <\/span><span class=\"NormalTextRun SCXW14107897 BCX8\">wish to thank<\/span><span class=\"NormalTextRun SCXW14107897 BCX8\"> Govind Kanshi (Principal PM Manager), <\/span><span class=\"NormalTextRun SCXW14107897 BCX8\">Abinav Rameesh<\/span> <span class=\"NormalTextRun SCXW14107897 BCX8\">(Principal PM Manager)<\/span><span class=\"NormalTextRun SCXW14107897 BCX8\">, <\/span><span class=\"NormalTextRun SCXW14107897 BCX8\">Kushagra Thapar (<\/span><span class=\"NormalTextRun SCXW14107897 BCX8\">Principal Engineering Manager), Annie Liang (S<\/span><span class=\"NormalTextRun SCXW14107897 BCX8\">r.<\/span><span class=\"NormalTextRun SCXW14107897 BCX8\"> Software Engineer) and Faiz Chachiya (S<\/span><span class=\"NormalTextRun SCXW14107897 BCX8\">r.<\/span><span class=\"NormalTextRun SCXW14107897 BCX8\"> Cloud Solution Architect<\/span><span class=\"NormalTextRun SCXW14107897 BCX8\">, CSU<\/span><span class=\"NormalTextRun SCXW14107897 BCX8\">)<\/span> <span class=\"NormalTextRun SCXW14107897 BCX8\">for their <\/span><span class=\"NormalTextRun SCXW14107897 BCX8\">valuable <\/span><span class=\"NormalTextRun SCXW14107897 BCX8\">insights, and contributions during the <\/span><span class=\"NormalTextRun SCXW14107897 BCX8\">Proof-of-Concept (<\/span><span class=\"NormalTextRun CommentStart CommentHighlightPipeRestV2 CommentHighlightRest SCXW14107897 BCX8\">PoC<\/span><span class=\"NormalTextRun CommentHighlightPipeRestV2 SCXW14107897 BCX8\">).<\/span><\/span><\/em><\/p>\n<h2><b><span data-contrast=\"auto\">Introduction<\/span><\/b><span data-ccp-props=\"{}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">Azure Cosmos DB is Microsoft\u2019s premier fully managed NoSQL database for modern app development. It is ideal for solutions including artificial intelligence, digital commerce, Internet of Things, booking management and other types of use cases. It offers single-digit millisecond response times, automatic and instant scalability along with guaranteed speed at any scale.<\/span><\/p>\n<p><span data-contrast=\"auto\">This blog post outlines the migration approach and Azure Cosmos DB for NoSQL Java SDK V4 micro-optimizations the team undertook while attempting to migrate an existing on-premises MongoDB database hosting a centralized Fraud Analytics platform to Azure Cosmos DB for NoSQL at a large Retail customer. The post highlights the PoC performance benchmark requirements, the challenges faced, and the resolution steps undertaken to achieve the PoC goals.<\/span><\/p>\n<h2><b><span data-contrast=\"auto\">Use case &amp; background<\/span><\/b><span data-ccp-props=\"{}\">\u00a0<\/span><\/h2>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">The customer is a retail corporation with a combination of both offline &amp; online ecommerce business models. With 100 Million+ unique visitors per month, the eCommerce architecture is a classic event-driven, loosely coupled, microservices architecture.<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"2\" data-aria-level=\"1\"><span data-contrast=\"auto\">The centralized fraud detection and analytics platform has a front-end capturing fire-and-forget transactional events which are then fed into a CQRS (Command and Query Responsibility Segregation) pattern-based dedicated databases. A MongoDB Cluster in an on-premises DC hosts multiple MongoDB databases.<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"3\" data-aria-level=\"1\"><span data-contrast=\"auto\">The Database for our PoC is referred to as Grade. Grade is essentially <\/span><b><span data-contrast=\"auto\">a time-series aggregate MongoDB database<\/span><\/b><span data-contrast=\"auto\"> capturing approximately 5,000+ parameters which are parameterized on specific user-defined rules.<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"4\" data-aria-level=\"1\"><span data-contrast=\"auto\">Pre-calculated aggregates are created and stored which include SUM, MIN, MAX, across a specific customer, email, address etc. across a period of minimum of 36 hours to maximum of 9 months.<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"5\" data-aria-level=\"1\"><span data-contrast=\"auto\">The purpose of this database is to answer pinpointed queries like, how much in USD did a customer named Subhasish spend on a specific SKU in the past 72 hours? The database is fine-tuned for a high-volume, sudden-burst scenario which are typical in eCommerce Holidays (e.g., Black Friday, Cyber Monday, Christmas &amp; New Year sales etc.).<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"6\" data-aria-level=\"1\"><span data-contrast=\"auto\">The primary query is to fetch various grade metrics history based on the indexed unique key. E.g., the following query returns all previous day&#8217;s records for a customer named Subhasish.\u202f<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<pre><span data-contrast=\"auto\">db.collection.query({key: 'Subhasish', date: '2023-07-11'})<\/span><span data-ccp-props=\"{&quot;335557856&quot;:16250356}\">\u00a0<\/span><\/pre>\n<ul style=\"list-style-type: circle;\">\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"3\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"7\" data-aria-level=\"1\"><span data-contrast=\"auto\">Peak throughput estimate during primary Holiday Season sales event:<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559685&quot;:360,&quot;335559739&quot;:0,&quot;335559740&quot;:240}\">\u00a0<\/span><span data-contrast=\"auto\">Reads: 100-120K\u202fops\/sec, <\/span><span data-contrast=\"auto\">Writes: 200-250K ops\/sec<\/span><span data-ccp-props=\"{&quot;335557856&quot;:16250356}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span class=\"TextRun SCXW133107709 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW133107709 BCX8\">A high-level block diagram of the architecture is <\/span><span class=\"NormalTextRun SCXW133107709 BCX8\">exhibited<\/span><span class=\"NormalTextRun SCXW133107709 BCX8\"> below<\/span><span class=\"NormalTextRun SCXW133107709 BCX8\">:<\/span><\/span><span class=\"EOP SCXW133107709 BCX8\" data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/GradeArchitecture-New.png\"><img decoding=\"async\" class=\"aligncenter wp-image-7632 size-large\" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/GradeArchitecture-New-1024x542.png\" alt=\"Image GradeArchitecture New\" width=\"640\" height=\"339\" srcset=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/GradeArchitecture-New-1024x542.png 1024w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/GradeArchitecture-New-300x159.png 300w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/GradeArchitecture-New-768x407.png 768w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/GradeArchitecture-New.png 1101w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<p><span class=\"TextRun SCXW87694837 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW87694837 BCX8\">The entire flow is <strong>real-time<\/strong> since <\/span><span class=\"NormalTextRun SCXW87694837 BCX8\">we cannot have aggregates getting calculated in real-time when<\/span><span class=\"NormalTextRun SCXW87694837 BCX8\"> the<\/span><span class=\"NormalTextRun SCXW87694837 BCX8\"> actual OLTP system is handling massive volume of transactions and <\/span><span class=\"NormalTextRun SCXW87694837 BCX8\">a customer\u2019s <\/span><span class=\"NormalTextRun SCXW87694837 BCX8\">order is pending confirmation in real-time since the fraud analytics system recommendation is yet to come through.<\/span><span class=\"NormalTextRun SCXW87694837 BCX8\"> In summary, this solution is <\/span><\/span><em><span class=\"TextRun SCXW87694837 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW87694837 BCX8\">highly <\/span><\/span><\/em><span class=\"TextRun SCXW87694837 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW87694837 BCX8\">latency sensitive<\/span><span class=\"NormalTextRun SCXW87694837 BCX8\">.<\/span><\/span><\/p>\n<h2><strong><span class=\"TextRun SCXW142690647 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW142690647 BCX8\">Performance <\/span><span class=\"NormalTextRun SCXW142690647 BCX8\">b<\/span><span class=\"NormalTextRun SCXW142690647 BCX8\">enchmarks<\/span><\/span><\/strong><\/h2>\n<p><span class=\"TextRun SCXW249802107 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW249802107 BCX8\">Table below captures the expected <\/span><span class=\"NormalTextRun SCXW249802107 BCX8\">latency in the <\/span><span class=\"NormalTextRun SCXW249802107 BCX8\">PoC<\/span><span class=\"NormalTextRun SCXW249802107 BCX8\"> post re-modeling from MongoDB to Azure Cosmos DB for NoSQL<\/span><span class=\"NormalTextRun SCXW249802107 BCX8\">:<\/span><\/span><\/p>\n<table style=\"border-collapse: collapse; width: 42.1393%; height: 185px;\">\n<tbody>\n<tr style=\"height: 54px;\">\n<td style=\"width: 12.59%; height: 54px;\">Operation<\/td>\n<td style=\"width: 12.0357%; height: 54px;\">Average<\/td>\n<td style=\"width: 12.0288%; height: 54px;\">P95<\/td>\n<td style=\"width: 12.896%; height: 54px;\">P99<\/td>\n<td style=\"width: 16.954%; height: 54px;\">App-entry point on VM<\/td>\n<td style=\"width: 19.4919%; height: 54px;\">Max Queries \/ sec\n(qps)<\/td>\n<\/tr>\n<tr style=\"height: 28px;\">\n<td style=\"width: 12.59%; height: 28px;\"><strong>Point-read<\/strong><\/td>\n<td style=\"width: 12.0357%; height: 28px;\"><strong>&lt; 3 ms<\/strong><\/td>\n<td style=\"width: 12.0288%; height: 28px;\"><strong>&lt; 10 ms<\/strong><\/td>\n<td style=\"width: 12.896%; height: 28px;\"><strong>&lt; 50 ms<\/strong><\/td>\n<td style=\"width: 16.954%; height: 28px;\"><strong>1<\/strong><\/td>\n<td style=\"width: 19.4919%; height: 28px;\"><strong>3,000<\/strong><\/td>\n<\/tr>\n<tr style=\"height: 28px;\">\n<td style=\"width: 12.59%; height: 28px;\">In-partition Query<\/td>\n<td style=\"width: 12.0357%; height: 28px;\">&lt; 7 ms<\/td>\n<td style=\"width: 12.0288%; height: 28px;\">&lt; 15 ms<\/td>\n<td style=\"width: 12.896%; height: 28px;\">&lt; 50 ms<\/td>\n<td style=\"width: 16.954%; height: 28px;\">1<\/td>\n<td style=\"width: 19.4919%; height: 28px;\">3,000<\/td>\n<\/tr>\n<tr style=\"height: 28px;\">\n<td style=\"width: 12.59%; height: 28px;\">Upsert<\/td>\n<td style=\"width: 12.0357%; height: 28px;\">&lt; 3 ms<\/td>\n<td style=\"width: 12.0288%; height: 28px;\">&lt; 5 ms<\/td>\n<td style=\"width: 12.896%; height: 28px;\">&lt; 10 ms<\/td>\n<td style=\"width: 16.954%; height: 28px;\">1<\/td>\n<td style=\"width: 19.4919%; height: 28px;\">3,000<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>&nbsp;<\/p>\n<p><span class=\"TextRun SCXW197098218 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW197098218 BCX8\">The crux of the PoC was to <\/span><\/span><span class=\"TextRun SCXW197098218 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW197098218 BCX8\">achieve a performance benchmark <\/span><span class=\"NormalTextRun SCXW197098218 BCX8\">of <\/span><\/span><strong><span class=\"TextRun SCXW197098218 BCX8\" lang=\"EN-US\" style=\"font-family: 'courier new', courier, monospace;\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW197098218 BCX8\">requestLatency<\/span><\/span><\/strong> <strong><span class=\"TextRun SCXW197098218 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW197098218 BCX8\">of <\/span><span class=\"NormalTextRun SCXW197098218 BCX8\">1 or 2 ms (i.e., &lt; 3 ms) for point-reads <\/span><span class=\"NormalTextRun SCXW197098218 BCX8\">of 0.<\/span><span class=\"NormalTextRun SCXW197098218 BCX8\">2<\/span><span class=\"NormalTextRun SCXW197098218 BCX8\"> KB item size <\/span><span class=\"NormalTextRun SCXW197098218 BCX8\">from a single App-entry point hosted on <\/span><span class=\"NormalTextRun SCXW197098218 BCX8\">an Azure virtual machine (VM) for a period of 1 hour consistently<\/span><span class=\"NormalTextRun SCXW197098218 BCX8\"> at 3,000 <\/span><span class=\"NormalTextRun SCXW197098218 BCX8\">qps<\/span><span class=\"NormalTextRun SCXW197098218 BCX8\">.<\/span><\/span><\/strong><span class=\"TextRun SCXW197098218 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW197098218 BCX8\"> The term <span style=\"font-family: 'courier new', courier, monospace;\">requestLatency<\/span> refers to how long a CRUD operation took. It represents a round trip time as observed by the client including retries (if any). <\/span><\/span><span class=\"TextRun SCXW197098218 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW197098218 BCX8\">The VM configuration <\/span><span class=\"NormalTextRun SCXW197098218 BCX8\">was 8 vCores \/ 27 GB.<\/span><\/span><\/p>\n<h2><span class=\"TextRun SCXW126922371 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW126922371 BCX8\">Re-modeling from MongoDB to Azure Cosmos DB for NoSQL<\/span><\/span><\/h2>\n<p><span data-contrast=\"auto\">The original MongoDB data design involved storing data for each unit in a separate MongoDB replica set. Each grade\/parameter is stored in a separate collection. <\/span><span style=\"font-family: 'courier new', courier, monospace;\" data-contrast=\"auto\">Time_bucket<\/span><span data-contrast=\"auto\"> is an integer which represents hour\/days\/etc. since the epoch. Primary access pattern is write-heavy for this database, with primary operation being <\/span><span style=\"font-family: 'courier new', courier, monospace;\" data-contrast=\"auto\">update()<\/span><span data-contrast=\"auto\"> with <\/span><span style=\"font-family: 'courier new', courier, monospace;\" data-contrast=\"auto\">$inc <\/span><span data-contrast=\"auto\">op. In MongoDB, you basically insert a new document if no match exists (upsert) with option upsert: true:\u00a0<\/span><span data-ccp-props=\"{&quot;335551550&quot;:6,&quot;335551620&quot;:6}\">\u00a0<\/span><\/p>\n<ol>\n<li data-leveltext=\"%1.\" data-font=\"Calibri Light,Times New Roman\" data-listid=\"5\" data-list-defn-props=\"{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">If document(s) match query criteria, <\/span><span style=\"font-family: 'courier new', courier, monospace;\" data-contrast=\"auto\">db.collection.update() <\/span><span data-contrast=\"auto\">performs an update.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"%1.\" data-font=\"Calibri Light,Times New Roman\" data-listid=\"5\" data-list-defn-props=\"{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">If no document matches the query criteria, <\/span><span style=\"font-family: 'courier new', courier, monospace;\" data-contrast=\"auto\">db.collection.update()<\/span><span data-contrast=\"auto\"> inserts a single document. And <\/span><span style=\"font-family: 'courier new', courier, monospace;\" data-contrast=\"auto\">$inc<\/span><span data-contrast=\"auto\"> update operator basically increments a field by a specified value.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ol>\n<p><span data-contrast=\"auto\">Sample item schema in MongoDB:<\/span><\/p>\n<pre>{ key,  time_bucket,  value }\r\n<em>Subh\u00a0\u00a0\u00a0 2023-05-01\u00a0\u00a0  11.51\r\n<\/em><em style=\"font-size: 14.4px; text-align: var(--bs-body-text-align);\">Subh\u00a0\u00a0\u00a0 2023-03-01\u00a0\u00a0\u00a0  9.00\r\n<\/em><em>Subh\u00a0\u00a0\u00a0 2023-06-04\u00a0\u00a0 200.88<\/em>\r\n<em>\u2026<\/em>\r\n<em>\u2026\r\n<\/em><em>Aayushi 2023-02-20\u00a0    5.00\r\n<\/em><em>Aayushi 2023-04-01\u00a0\u00a0  15.00<\/em><\/pre>\n<p>In Azure Cosmos DB for NoSQL, all Grade collections for a given time unit in MongoDB basically translate to a single container in CosmosDB. A time bucket record basically has a <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/nosql\/synthetic-partition-keys\" target=\"_blank\" rel=\"noopener\">synthetic partition key<\/a> (pk) which is created by concatenating multiple properties of an item. This ensures we get an even distribution of data for a given Grade type, as well as across all Grade types. A sample JSON schema for an item:<\/p>\n<pre>    pk: &lt; Domain | GradeName | TimUnit | Key &gt;\r\n    id: &lt; TimUnit &gt;\r\nbucket: &lt; Date &gt;\r\n   val: &lt; number &gt;\r\n \u00a0\u00a0ttl: &lt; secs to expire &gt;<\/pre>\n<p>A sample input document using the above schema:<\/p>\n<pre> \u00a0\u00a0\u00a0pk: ssp.com|cust_ord|Day|125636751&gt;\r\n \u00a0\u00a0\u00a0id:\u00a0 18813\r\nbucket:\u00a0 &lt;Date&gt;\r\n   val: 192.33\r\n   ttl: 120<\/pre>\n<p>In our example above, <span style=\"font-family: 'courier new', courier, monospace;\">com<\/span> is domain, <span style=\"font-family: 'courier new', courier, monospace;\">cust_ord<\/span> is GradeName, <span style=\"font-family: 'courier new', courier, monospace;\">Day<\/span> is the time-unit bucket, and finally, <span style=\"font-family: 'courier new', courier, monospace;\">125636751<\/span> is the key identifying a specific customer. <span style=\"font-family: 'courier new', courier, monospace;\">Id<\/span> represents time-bucket represented as an integer value of time units since epoch. An important point to note is that all records for a given key will land in the same logical partition. Thus, for a given grade parameter say, <span style=\"font-family: 'courier new', courier, monospace;\">cust_ord<\/span>, all records for this specific customer <span style=\"font-family: 'courier new', courier, monospace;\">125636751<\/span> will reside on the same logical partition.<\/p>\n<p>Partition key model in Azure Cosmos DB for NoSQL<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/pk-model.png\"><img decoding=\"async\" class=\"size-large wp-image-7602 aligncenter\" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/pk-model-1024x361.png\" alt=\"Partition Key model\" width=\"640\" height=\"226\" srcset=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/pk-model-1024x361.png 1024w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/pk-model-300x106.png 300w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/pk-model-768x271.png 768w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/pk-model-1536x542.png 1536w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/pk-model.png 1826w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<ul>\n<li>Each Key resides in a unique partition.<\/li>\n<li>Each <span style=\"font-family: 'courier new', courier, monospace;\">[GradeN]<\/span> and associated bucket records reside within the partition.<\/li>\n<li>A single query fetches all grades and associated bucket records in a partition.<\/li>\n<\/ul>\n<h2><strong>Functional Tests<\/strong><\/h2>\n<p>For our test purposes, we used the following setup:<\/p>\n<ul>\n<li>An Azure Cosmos DB for NoSQL container with a throughput of 20,000 RU\/s<\/li>\n<li>Item size of 0.2 KB<\/li>\n<li>Operations primarily to be tested: point read and upsert<\/li>\n<li>5,000 async writes including 40% creates and 60% increments (using <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/partial-document-update\" target=\"_blank\" rel=\"noopener\">Patch API<\/a>)<\/li>\n<li>10 threads<\/li>\n<li>Java SDK v4 4.36.0 \/\/ Replace with the latest version in your test<\/li>\n<li>Account configured with Direct Connectivity mode, Session Consistency<\/li>\n<li>Test VM: South Central US<\/li>\n<li>Azure Region hosting Azure Cosmos DB: South Central US<\/li>\n<li>GitHub repo with sample schema, settings and load test code in Java can be accessed <a href=\"https:\/\/github.com\/SubhMSFT\/CosmosDBLoadTestCode\" target=\"_blank\" rel=\"noopener\">here<\/a><\/li>\n<\/ul>\n<p><div class=\"alert alert-success\">Recommended best practice is to always use the latest Java SDK V4. For your tests, replace Java SDK version with the latest version.<\/div><\/p>\n<h2><strong>Challenges <\/strong><\/h2>\n<h3>Challenge #1: P99 requestLatency on create\/patch was 300+ ms at 500 qps: How to diagnose and troubleshoot?<\/h3>\n<p><strong>Resolution:<\/strong> Recommended practice is to proactively leverage Azure Cosmos DB for NoSQL Java SDK V4 client-side Diagnostics during dev\/test stage of a PoC. Azure Cosmos DB for NoSQL Java SDK V4 enables you to extract and represent information related to database, container, item, and query operations using the\u00a0<a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/nosql\/troubleshoot-java-sdk-v4?tabs=sync\" target=\"_blank\" rel=\"noopener\">Diagnostics\u00a0property<\/a> in the response object of a request. This property contains information such as CPU and memory levels on the compute host for the SDK instance as well as service-side metrics covering nearly every aspect of the execution of your request. The JSON output could be used effectively to undertake proactive custom conditional client-side monitoring &amp; in-depth troubleshooting of <a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/java-sdk-v4-diagnostics\/\" target=\"_blank\" rel=\"noopener\">commonly found client-side issues<\/a>. In our PoC, diagnostics provided visibility into transport layer events, a sample plot is as exhibited below:<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge1-Image1.png\"><img decoding=\"async\" class=\"size-full wp-image-7608 aligncenter\" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge1-Image1.png\" alt=\"Azure Cosmos DB for NoSQL Java SDK V4 Diagnostics output plot\" width=\"833\" height=\"475\" srcset=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge1-Image1.png 833w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge1-Image1-300x171.png 300w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge1-Image1-768x438.png 768w\" sizes=\"(max-width: 833px) 100vw, 833px\" \/><\/a><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge1-Image2.png\"><img decoding=\"async\" class=\"size-full wp-image-7609 aligncenter\" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge1-Image2.png\" alt=\"Image Challenge1 Image2\" width=\"841\" height=\"283\" srcset=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge1-Image2.png 841w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge1-Image2-300x101.png 300w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge1-Image2-768x258.png 768w\" sizes=\"(max-width: 841px) 100vw, 841px\" \/><\/a><\/p>\n<p>Additionally, review of <span style=\"font-family: 'courier new', courier, monospace;\">systemCpuLoad<\/span> property in <span style=\"font-family: 'courier new', courier, monospace;\">systemInformation<\/span> section of the diagnostics JSON output showed consistently high CPU (75%+) during creates. Firing 100\u2019s of create calls (async API) in a loop non-stop generated high CPU load.<\/p>\n<pre><span style=\"font-family: 'courier new', courier, monospace;\">\"systemCpuLoad\":\"<\/span>\r\n<span style=\"font-family: 'courier new', courier, monospace;\"> \u00a0\u00a0 (2022-05-02T07:27:59.495855600Z 79.0%),<\/span>\r\n<span style=\"font-family: 'courier new', courier, monospace;\"> \u00a0\u00a0 (2022-05-02T07:28:04.489517300Z 75.0%),<\/span>\r\n<span style=\"font-family: 'courier new', courier, monospace;\"> \u00a0\u00a0 (2022-05-02T07:28:09.495177300Z 78.0%),<\/span>\r\n<span style=\"font-family: 'courier new', courier, monospace;\"> \u00a0\u00a0 (2022-05-02T07:28:14.495187100Z 79.0%).<\/span><\/pre>\n<p>We concluded that executing 100\u2019s of create calls (async API) in a loop non-stop is generating too much pressure on the Java SDK V4 underlying Reactor library. We added a 1-ms delay between each async create call in the loop. This improved the CPU load on the client-machine, and we observed single-digit ms on create and patch calls.<\/p>\n<p>Below stats are from 5000 calls to API <span style=\"font-family: 'courier new', courier, monospace;\">createOrIncrement<\/span> \u2013 which first attempts to create an item and if it fails, does a patch \/ increment operation:<\/p>\n<pre>createOrIncrement (ms)      [count: 5000, avg:\u00a0\u00a0 9 | min:\u00a0 4, max:\u00a0 421 | med:\u00a0\u00a0 8, 95%:\u00a0 10]\r\n\r\n---------------------------------------------------------------------------------------------\r\n\r\n\u00a0 |-create (ms)\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0[count: 1814, avg:\u00a0\u00a0 7 | min:\u00a0 4, max:\u00a0 385 | med:\u00a0\u00a0 6, 95%:\u00a0\u00a0 8]\r\n\r\n---------------------------------------------------------------------------------------------\r\n  |-create conflict (ms)\u00a0\u00a0\u00a0 [count: 3186, avg:\u00a0\u00a0 2 | min:\u00a0 1, max:\u00a0 328 | med:\u00a0\u00a0 2, 95%:\u00a0\u00a0 3]\r\n\r\n  |-patch (ms)\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0[count: 3186, avg:\u00a0\u00a0 6 | min:\u00a0 5, max:\u00a0\u00a0 65 | med:\u00a0\u00a0 6, 95%:\u00a0\u00a0 7]<\/pre>\n<p><div class=\"alert alert-success\">Learning #1: Pro-actively leverage Azure Cosmos DB for NoSQL Java SDK V4 Diagnostics conditional logging for diagnosing latency related issues early-on into development.<\/div><\/p>\n<h3>Challenge #2: At 900 qps, we received high channelAcquisitionStarted times in creates. This negatively impacted our ability to push queries further.<\/h3>\n<p><strong>Resolution:<\/strong> In an attempt to scale from 500 qps to 1,000 qps and determine max throughput that is achievable by 1 VM, we received high <span style=\"font-family: 'courier new', courier, monospace;\">channelAcquisitionStarted<\/span> times at around 900 qps as exhibited below.<\/p>\n<p>The <span style=\"font-family: 'courier new', courier, monospace;\">transportRequestTimeline<\/span> section of the Diagnostics JSON output provides essential information on the specific transport layer events since the CRUD request was fired. The <span style=\"font-family: 'courier new', courier, monospace;\">channelAcquisitionStarted<\/span> event denotes we have successfully queued the request, and now all we need is for us to acquire a channel which will enable to send the request to the actual endpoint. The endpoint here refers to the full rntbd (Microsoft TCP layered stack) FQDN address including partition &amp; replica IDs. A channel is a connection from your client VM to Azure Cosmos DB machine which hosts the replica to which this request is now going to be transported.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge2-Image1.png\"><img decoding=\"async\" class=\"aligncenter wp-image-7615 size-large\" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge2-Image1-1024x445.png\" alt=\"Image Challenge2 Image1\" width=\"640\" height=\"278\" \/><\/a><\/p>\n<p>We customized <span style=\"font-family: 'courier new', courier, monospace;\">connCfg<\/span> rntbd network connectivity parameters for Azure Cosmos DB via Azure Cosmos DB Java SDK V4.<\/p>\n<pre>\"connCfg\":{\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \"rntbd\":\"(cto:PT5S, nrto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30, cer:true)\",\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\r\n \u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0 \u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\u2026\r\n \u00a0\u00a0\u00a0\u00a0    },<\/pre>\n<p>Read this <a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/java-sdk-v4-diagnostics\/#scenario-3-point-reads-exhibiting-high-latency-in-a-high-volume-sudden-burst-scenario\" target=\"_blank\" rel=\"noopener\">blog post<\/a> for a detailed analysis of the individual parameters and suggested <span style=\"font-family: 'courier new', courier, monospace;\">connCfg<\/span> values based on different read and write optimized eCommerce use-cases.<\/p>\n<p>In the above, <span style=\"font-family: 'courier new', courier, monospace;\">cto<\/span> stands for <span style=\"font-family: 'courier new', courier, monospace;\">ConnectionTimeout<\/span> duration on the rntbd TCP stack for a given CRUD request (in our use-case scenario, it is for the create operation). This signifies that in case you are facing any connectivity issues, you are instructing the TCP stack to retry that specific request after a customized time duration interval. By default, <span style=\"font-family: 'courier new', courier, monospace;\">cto<\/span> is set to 5 seconds by the Java SDK V4. For specific use-cases you can choose to customize it. E.g., if you are running an eCommerce site which is expecting to observe a high volume of peak requests during a Black Friday event, you can choose to customize the <span style=\"font-family: 'courier new', courier, monospace;\">cto<\/span> values based on the query patterns in your code. Here are some example values we\u2019ve observed customers using:<\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ol>\n<li>Read heavy app primarily doing point-reads: 600 ms<\/li>\n<li>Read heavy app primarily doing in-partition queries: 600 ms<\/li>\n<li>An app primarily executing stored procedures: 1 sec<\/li>\n<li>Write heavy app: 600 ms<\/li>\n<li>An app which does cross-partition queries: 5 sec<\/li>\n<li>An app which primarily does Cross Region calls: 1 Sec<\/li>\n<\/ol>\n<\/li>\n<\/ul>\n<p>Initial setup was changed from <span style=\"font-family: 'courier new', courier, monospace;\">cto<\/span> = 5 seconds<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge2-Image2.png\"><img decoding=\"async\" class=\"wp-image-7616 aligncenter\" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge2-Image2.png\" alt=\"connCfg values in Azure Cosmos DB Diagnostics\" width=\"765\" height=\"94\" srcset=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge2-Image2.png 781w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge2-Image2-300x37.png 300w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge2-Image2-768x94.png 768w\" sizes=\"(max-width: 765px) 100vw, 765px\" \/><\/a><\/p>\n<p>To <span style=\"font-family: 'courier new', courier, monospace;\">cto<\/span> = 600 ms (0.6 seconds)<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge2-Image3.png\"><img decoding=\"async\" class=\"wp-image-7617 aligncenter\" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge2-Image3.png\" alt=\"Image Challenge2 Image3\" width=\"762\" height=\"93\" srcset=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge2-Image3.png 746w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge2-Image3-300x37.png 300w\" sizes=\"(max-width: 762px) 100vw, 762px\" \/><\/a><\/p>\n<pre><span style=\"color: #339966;\">\/* Exercise caution in production environments.<\/span>\r\n<span style=\"color: #339966;\">\/* Reach out to Microsoft for specific guidance.<\/span>\r\nDirectConnectionConfig directConnectionConfig = DirectConnectionConfig.getDefaultConfig();\r\ndirectConnectionConfig.setConnectTimeout(Duration.ofMillis(600));\r\ndirectConnectionConfig.setNetworkRequestTimeout(Duration.ofSeconds(5));\r\ndirectConnectionConfig.setIdleConnectionTimeout(Duration.ofSeconds(0));\r\ndirectConnectionConfig.setIdleEndpointTimeout(Duration.ofHours(1));\r\ndirectConnectionConfig.setMaxConnectionsPerEndpoint(350);\r\nclient = new CosmosClientBuilder()\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .endpoint(AccountSettings.HOST)\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .key(AccountSettings.MASTER_KEY)\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .consistencyLevel(ConsistencyLevel.SESSION)\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .contentResponseOnWriteEnabled(false)\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .directMode(directConnectionConfig)\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .buildAsyncClient();<\/pre>\n<h5><strong><u>Additional Tip:<\/u><\/strong><\/h5>\n<p>In addition to the above, you can leverage an <strong>end-to-end timeout policy feature<\/strong> in Java SDK V4. An end-to-end timeout policy is specifically catered to solve challenges related to tail latency which effectively cause availability to drop significantly since requests are not being cancelled. You provide a timeout value that covers the whole execution of any request, including requests that span multiple partitions. Ongoing requests will be cancelled if the timeout setting is reached. The timeout duration can be set on <span style=\"font-family: 'courier new', courier, monospace;\">CosmosItemRequestOptions<\/span>. The options can then be passed to any request sent to Azure Cosmos DB. Read this <a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/latest-nosql-java-ecosystem-updates-2023-q1-q2\/#end-to-end-timeout-policy\" target=\"_blank\" rel=\"noopener\">blog post for more information and sample code for enabling this feature<\/a>.<\/p>\n<p>The above two changes lowered our <span style=\"font-family: 'courier new', courier, monospace;\">channelAcquisitionStarted<\/span> times for bulk of our queries, thereby positively impacting the round-trip latency.<\/p>\n<p><div class=\"alert alert-success\">Learning #2: Review your use-case and customize the connCfg rntbd network connectivity parameters in Azure Cosmos DB using Java SDK V4.<\/div><\/p>\n<h3>Challenge #3: At 1,500 qps, we received high channelAcquisitionStarted times in creates. This negatively impacted our ability to push queries further.<\/h3>\n<p><strong>Resolution:<\/strong> In an attempt to further optimize and lower <span style=\"font-family: 'courier new', courier, monospace;\">channelAcquisitionStarted<\/span> times at around 1,300-1,500 qps, we upgraded SDK from 4.36.0 -&gt; 4.42.0 <em>(the latest stable version of the Java SDK available during the time of the PoC)<\/em> which allowed us to use <span style=\"font-family: 'courier new', courier, monospace;\"><a href=\"https:\/\/learn.microsoft.com\/java\/api\/com.azure.cosmos.cosmosclientbuilder?view=azure-java-stable#com-azure-cosmos-cosmosclientbuilder-openconnectionsandinitcaches(com-azure-cosmos-cosmoscontainerproactiveinitconfig)\" target=\"_blank\" rel=\"noopener\">openConnectionsAndInitCaches<\/a><\/span> API. This API allows us to set <span style=\"font-family: 'courier new', courier, monospace;\"><a href=\"https:\/\/learn.microsoft.com\/java\/api\/com.azure.cosmos.cosmoscontainerproactiveinitconfig?view=azure-java-stable\" target=\"_blank\" rel=\"noopener\">CosmosContainerProactiveInitConfig<\/a><\/span> value for implementing <strong><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/latest-nosql-java-ecosystem-updates-2023-q1-q2\/#java-sdk-proactive-connection-management\" target=\"_blank\" rel=\"noopener\">proactive connection management<\/a><\/strong>. This feature, added to the Java SDK in Feb 2023, enables warming up connections and caches for containers for both the current and read region and a pre-defined number of preferred remote regions, as opposed to just the current region. This feature is especially useful in a few scenarios wherein improving the tail latency in cross-region failover scenarios and reducing overall latency for writes in a multi-region scenario wherein only a single write region is configured. Read this <a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/latest-nosql-java-ecosystem-updates-2023-q1-q2\/#java-sdk-proactive-connection-management\" target=\"_blank\" rel=\"noopener\">blog post for more information and sample code for enabling this feature<\/a>.<\/p>\n<p>A point noteworthy is that the proactive connection regions are a subset of the preferred regions configured through the\u00a0<span style=\"font-family: 'courier new', courier, monospace;\"><a href=\"https:\/\/learn.microsoft.com\/java\/api\/com.azure.cosmos.cosmosclientbuilder?view=azure-java-stable\" target=\"_blank\" rel=\"noopener\">CosmosClientBuilder<\/a><\/span>. The first\u00a0<span style=\"font-family: 'courier new', courier, monospace;\"><a href=\"https:\/\/learn.microsoft.com\/java\/api\/com.azure.cosmos.cosmoscontainerproactiveinitconfig?view=azure-java-stable#com-azure-cosmos-cosmoscontainerproactiveinitconfig-getproactiveconnectionregionscount()\" target=\"_blank\" rel=\"noopener\">getProactiveConnectionRegionsCount()<\/a><\/span>\u00a0read regions from preferred regions are picked.<\/p>\n<p>This change assisted in lowering the request latency from earlier observed numbers and solved <span style=\"font-family: 'courier new', courier, monospace;\">channelAcquisitionStarted<\/span> <span style=\"font-family: 'courier new', courier, monospace;\">durationInMilliSecs<\/span> for a large batch of queries.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge3-Image1.png\"><img decoding=\"async\" class=\"aligncenter wp-image-7620 size-full\" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge3-Image1.png\" alt=\"Image Challenge3 Image1\" width=\"1091\" height=\"666\" srcset=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge3-Image1.png 1091w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge3-Image1-300x183.png 300w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge3-Image1-1024x625.png 1024w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge3-Image1-768x469.png 768w\" sizes=\"(max-width: 1091px) 100vw, 1091px\" \/><\/a><\/p>\n<p>E.g., one of the worst performing queries at 1,142.0 ms reduced to 0.0 ms post incorporation of this change.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge3-Image2.png\"><img decoding=\"async\" class=\"aligncenter wp-image-7623 size-large\" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge3-Image2-1024x130.png\" alt=\"Image Challenge3 Image2\" width=\"640\" height=\"81\" srcset=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge3-Image2-1024x130.png 1024w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge3-Image2-300x38.png 300w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge3-Image2-768x97.png 768w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge3-Image2.png 1198w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<h5><strong><u>Additional Tip<\/u><\/strong><strong><u>:<\/u><\/strong><\/h5>\n<p>A new <strong>threshold-based parallel processing capability<\/strong> in the Java SDK has also been added, which can be activated when creating an end-to-end timeout policy explained in one of the earlier sections, to improve tail latency and availability even further. When enabled, parallel executions of the same request (read requests only) will be sent to secondary regions, where the request that responds fastest is the one that is accepted.<\/p>\n<p>Assume you have three regions set as <span style=\"font-family: 'courier new', courier, monospace;\">preferredRegions<\/span> in\u00a0<span style=\"font-family: 'courier new', courier, monospace;\"><a href=\"https:\/\/aka.ms\/CosmosJavaPreferredRegions\" target=\"_blank\" rel=\"noopener\">CosmosClientBuilder<\/a><\/span>, in the following order: East\u00a0US,\u00a0East\u00a0US\u00a02,\u00a0South Central US.<\/p>\n<p>Assume the following values for speculation threshold and threshold step:<\/p>\n<ul>\n<li>int threshold = 500<\/li>\n<li>int thresholdStep = 100;<\/li>\n<\/ul>\n<ol>\n<li>At time T1, a request to\u00a0East\u00a0US\u00a0is made.<\/li>\n<li>If there is no response in 500ms, a request to\u00a0East\u00a0US\u00a02\u00a0is made.<\/li>\n<li>If there is no response at 500+100ms, a request to\u00a0South Central US\u00a0is made.<\/li>\n<\/ol>\n<p>Read this <a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/latest-nosql-java-ecosystem-updates-2023-q1-q2\/#threshold-based-availability-optimization\" target=\"_blank\" rel=\"noopener\">blog post for more information and sample code for enabling this feature<\/a>.<\/p>\n<p><div class=\"alert alert-success\">Learning #3: Leverage Azure Cosmos DB for NoSQL Java SDK V4 specific properties for query optimization and improving tail latencies.<\/div><\/p>\n<h3>Challenge #4: How to scale from 1,500 qps to 3,000 qps for achieving 2 ms requestLatency using Java SDK V4?<\/h3>\n<p><strong>Resolution:<\/strong> The above-mentioned three learnings enabled us to lower P99 <span style=\"font-family: 'courier new', courier, monospace;\">requestLatency<\/span> from 300 ms+ to avg. 50\u00a0ms\u00a0consistently for both point-reads and in-partition queries. To optimize further to reach 1 or 2 ms average latency for point-reads, we undertook two micro-optimization steps:<\/p>\n<p><strong>Step1:<\/strong> We undertook an application code analysis. Azure Cosmos DB for NoSQL Java SDK V4 is based on <a href=\"https:\/\/projectreactor.io\/\" target=\"_blank\" rel=\"noopener\">Project Reactor<\/a> and <a href=\"https:\/\/netty.io\/\" target=\"_blank\" rel=\"noopener\">Project Netty<\/a> library. Designed for reactive programming, the Async API sends requests to Azure Cosmos DB for NoSQL using Reactor Netty with async I\/O at the OS level &#8211; no block waiting for the responses. This allows us to push out as many requests per second as your system hardware and your provisioned throughput allows. The reactive streams allow asynchronous stream processing by scheduling tasks on worker threads. We did a comparative performance code analysis on the same Flux and observed performance difference between the two available Project Reactor Schedulers: <span style=\"font-family: 'courier new', courier, monospace;\">subscribeOn<\/span> and <span style=\"font-family: 'courier new', courier, monospace;\">publishOn<\/span>.<\/p>\n<p>We switched from default <span style=\"font-family: 'courier new', courier, monospace;\">.subscribeOn<\/span> to\u00a0<span style=\"font-family: 'courier new', courier, monospace;\">.publishOn<\/span> owing to <span style=\"font-family: 'courier new', courier, monospace;\">publishOn<\/span> offsetting the blocking work on a separate thread. A sample transformation is exhibited below:<\/p>\n<pre>public static void createRecordAsync(final CosmosAsyncContainer container, final String pkey, final String id) {\r\n \u00a0\u00a0 try {\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \/\/ \u2026\u2026 \/\/\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 container\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .createItem(day, new PartitionKey(pkey), options)\r\n                <span style=\"color: #008080;\">\/\/ .subscribeOn(Schedulers.boundedElastic())<\/span>\r\n                .publishOn(Schedulers.boundedElastic())\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .doOnSuccess(response -&gt; {\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \/\/ \u2026\u2026 \/\/\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 double requestCharge = response.getRequestCharge();\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Duration requestLatency = response.getDuration();\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 System.out.println(response.getDiagnostics().toString());\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \/\/System.out.println(\"\\n\\n\\n\");\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 })\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .doOnError(error -&gt; {\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 error.printStackTrace();\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 })\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .subscribe();\r\n \u00a0\u00a0 } catch (CosmosException ce) {\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ce.printStackTrace();\r\n \u00a0\u00a0 }\r\n}<\/pre>\n<p>Since our use-case is predominantly targeted towards optimizing non-blocking calls, using <span style=\"font-family: 'courier new', courier, monospace;\">subscribeOn<\/span>\u00a0means running the initial source emission e.g.,\u00a0<span style=\"font-family: 'courier new', courier, monospace;\">subscribe()<\/span>, <span style=\"font-family: 'courier new', courier, monospace;\">onSubscribe()<\/span> and <span style=\"font-family: 'courier new', courier, monospace;\">request()<\/span>\u00a0on a specified scheduler worker. This is also the same for any subsequent operations, e.g., <span style=\"font-family: 'courier new', courier, monospace;\">onNext\/onError\/onComplete<\/span>, <span style=\"font-family: 'courier new', courier, monospace;\">map<\/span> etc.\u00a0and no matter the position of <span style=\"font-family: 'courier new', courier, monospace;\">subscribeOn()<\/span>, this behavior would happen. In summary, <span style=\"font-family: 'courier new', courier, monospace;\">subscribeOn<\/span> forces the source emission to use specific Schedulers.<\/p>\n<p>On the other hand, using <span style=\"font-family: 'courier new', courier, monospace;\">publishOn<\/span> changes Schedulers for all the downstream operations in the pipeline. This change of task execution context for the operations in the downstream using <span style=\"font-family: 'courier new', courier, monospace;\"><strong>.publishOn(Schedulers.boundedElastic())<\/strong><\/span> uses a thread pool containing\u00a010 * number of CPU cores\u00a0of your app entry point (in our use-case, Azure VM).\u00a0This is an excellent choice for optimizing IO operations or any blocking call.<\/p>\n<pre>Flux&lt;Integer&gt; flux = Flux.range(0, 2)\r\n \u00a0 \u00a0\u00a0\u00a0\u00a0.publishOn(Schedulers.boundedElastic());<\/pre>\n<p>This solved our on-going latencies and dropped P99 &lt; 10 ms.<\/p>\n<p><strong>Step2:<\/strong> Our application code was using the Scatter Gather pattern in Java that allows you to route messages to a number of dynamically specified recipients and re-aggregate the responses back into a single message. We replaced Scatter method with <span style=\"font-family: 'courier new', courier, monospace;\">subscribe()<\/span> when issuing each query within the loop. Instead of collecting all 1,500 query fluxes and firing at once using Scatter Gather pattern, we made the following change:<\/p>\n<pre>\/\/ Initially, we were firing all fluxes at about the same time and collecting results (Java Scatter-Gather pattern)\r\nFlux.merge(fluxes).collectList().subscribe(\/\/your method here);<\/pre>\n<pre>\/\/ We changed to:\r\nFlux.merge(mono).collectList().subscribe(\/\/your method here);<\/pre>\n<p>Here, the merge\u00a0function executes a merging of the data from\u00a0Publisher\u00a0sequences contained in an array into an interleaved merged sequence. We do a <span style=\"font-family: 'courier new', courier, monospace;\"><a href=\"https:\/\/projectreactor.io\/docs\/core\/release\/api\/reactor\/core\/publisher\/Flux.html#collectList--\" target=\"_blank\" rel=\"noopener\">collectList()<\/a><\/span> which accumulates the sequence emitted by this Flux into a List that is emitted by the resulting Mono when this sequence completes.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge4-Image1.png\"><img decoding=\"async\" class=\"aligncenter wp-image-7624 \" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge4-Image1.png\" alt=\"Image Challenge4 Image1\" width=\"785\" height=\"463\" srcset=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge4-Image1.png 956w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge4-Image1-300x177.png 300w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2024\/02\/Challenge4-Image1-768x453.png 768w\" sizes=\"(max-width: 785px) 100vw, 785px\" \/><\/a><\/p>\n<p>The above-mentioned micro-optimizations resolved our on-going latencies and dropped avg. latency for point-reads &lt; 3 ms. Additionally, the performance held consistently over a duration of a few hours, when we scaled from our initial 1,500 qps to 3,000 qps. Finally, we subjected the Test setup for spike testing by scaling the same set of queries (point-read, query and upsert) for max qps to 50,000 (across multiple entry-points) in a sporadic fashion to mimic real-life eCommerce sales event scenarios (e.g., Black Friday sales event). The above micro-optimizations allowed us to achieve similar results thereby satisfying the desired outcome in the tests.<\/p>\n<p><div class=\"alert alert-success\">Learning #4: Leverage appropriate Reactive non-blocking patterns in code to reduce tail latency and further optimize execution latency down to a single-digit millisecond.<\/div><\/p>\n<h2><strong>Summary of Learnings<\/strong><\/h2>\n<ol>\n<li><strong>Learning #1:<\/strong> Pro-actively leverage Azure Cosmos DB for NoSQL Java SDK V4 Diagnostics conditional logging for diagnosing latency related issues early-on into development.<\/li>\n<li><strong>Learning #2:<\/strong> Review your use-case and customize the connCfg rntbd network connectivity parameters in Azure Cosmos DB using Java SDK V4.<\/li>\n<li><strong>Learning #3:<\/strong> Leverage Azure Cosmos DB for NoSQL Java SDK V4 specific properties for query optimization and improving tail latencies.<\/li>\n<li><strong>Learning #4:<\/strong> Leverage appropriate Reactive non-blocking patterns in code to reduce tail latency and further optimize execution latency down to a single-digit millisecond.<\/li>\n<\/ol>\n<h2><strong>Conclusion<\/strong><\/h2>\n<p>Azure Cosmos DB for NoSQL offers single-digit millisecond response times, automatic and instant scalability along with guaranteed speed at any scale. Best practices for using Azure Cosmos DB for NoSQL Java SDK V4 allow you to improve your latency, availability and boost overall performance. Leveraging the latest Java SDK V4 coupled with specific SDK level optimizations is advisable if you have high performance and low latency scenarios as covered in our blog post. Do reach out to the Azure Cosmos DB Product team for specific insights into your use-case scenario.<\/p>\n<p>Let us know your thoughts in the comments section below.<\/p>\n<h2><strong>Where to learn more.<\/strong><\/h2>\n<p>Explore the following links for further information:<\/p>\n<ul>\n<li>Azure Cosmos DB for NoSQL Java SDK <a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-java\/tree\/main\" target=\"_blank\" rel=\"noopener\">GitHub repository<\/a><\/li>\n<li>Azure Cosmos DB for NoSQL Java SDK V4 <a href=\"https:\/\/github.com\/Azure-Samples\/azure-cosmos-java-sql-api-samples\/blob\/main\/reactor-pattern-guide.md\" target=\"_blank\" rel=\"noopener\">Reactor pattern guide<\/a><\/li>\n<li>Azure Cosmos DB for NoSQL Java SDK V4 <a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-java\/blob\/main\/sdk\/cosmos\/azure-cosmos\/CHANGELOG.md\" target=\"_blank\" rel=\"noopener\">CHANGE LOG<\/a><\/li>\n<li>Azure Cosmos DB for NoSQL Java SDK <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/nosql\/best-practice-java\" target=\"_blank\" rel=\"noopener\">Best Practices<\/a><\/li>\n<li>Azure Cosmos DB for NoSQL SDK v4 <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/nosql\/troubleshoot-java-sdk-v4?tabs=sync\" target=\"_blank\" rel=\"noopener\">Troubleshooting<\/a><\/li>\n<li><a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/nosql\/performance-tips-java-sdk-v4?tabs=api-async\" target=\"_blank\" rel=\"noopener\">Performance Tips<\/a> for Azure Cosmos for NoSQL Java SDK v4<\/li>\n<li><a href=\"https:\/\/aka.ms\/trycosmosdb\" target=\"_blank\" rel=\"noopener\">Try Azure Cosmos DB for free<\/a><\/li>\n<li>Azure Cosmos DB <a href=\"https:\/\/www.youtube.com\/c\/AzureCosmosDB\/featured\" target=\"_blank\" rel=\"noopener\">Live TV &#8211; YouTube Channel<\/a><\/li>\n<li>Azure Cosmos DB <a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/\" target=\"_blank\" rel=\"noopener\">Blog<\/a>. Subscribe for latest news, updates &amp; technical insights.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>The authors wish to thank Govind Kanshi (Principal PM Manager), Abinav Rameesh (Principal PM Manager), Kushagra Thapar (Principal Engineering Manager), Annie Liang (Sr. Software Engineer) and Faiz Chachiya (Sr. Cloud Solution Architect, CSU) for their valuable insights, and contributions during the Proof-of-Concept (PoC). Introduction\u00a0 Azure Cosmos DB is Microsoft\u2019s premier fully managed NoSQL database for [&hellip;]<\/p>\n","protected":false},"author":93861,"featured_media":7620,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[14,1611,1828,643,996,19],"tags":[499,1898,1806,1736,1841,286,1246,1797,1899,1900],"class_list":["post-7587","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-core-sql-api","category-data-architecture","category-data-modeling","category-java-sdk","category-migration","category-tips-and-tricks","tag-azure-cosmos-db","tag-azure-cosmos-db-for-nosql-load-testing","tag-cosmos-db-java-sdk","tag-data-modeling","tag-java-sdk-v4","tag-migration","tag-mongodb","tag-performance","tag-project-reactor","tag-reactive-programming"],"acf":[],"blog_post_summary":"<p>The authors wish to thank Govind Kanshi (Principal PM Manager), Abinav Rameesh (Principal PM Manager), Kushagra Thapar (Principal Engineering Manager), Annie Liang (Sr. Software Engineer) and Faiz Chachiya (Sr. Cloud Solution Architect, CSU) for their valuable insights, and contributions during the Proof-of-Concept (PoC). Introduction\u00a0 Azure Cosmos DB is Microsoft\u2019s premier fully managed NoSQL database for [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/posts\/7587","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/users\/93861"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/comments?post=7587"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/posts\/7587\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/media\/7620"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/media?parent=7587"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/categories?post=7587"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/tags?post=7587"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}