{"id":7358,"date":"2024-01-08T07:00:17","date_gmt":"2024-01-08T15:00:17","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cosmosdb\/?p=7358"},"modified":"2024-02-08T21:16:36","modified_gmt":"2024-02-09T05:16:36","slug":"best-practices-for-re-modeling-graph-database-to-azure-cosmos-db-for-nosql","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cosmosdb\/best-practices-for-re-modeling-graph-database-to-azure-cosmos-db-for-nosql\/","title":{"rendered":"Best Practices for Re-modeling Graph database to Azure Cosmos DB for NoSQL"},"content":{"rendered":"

Introduction<\/strong><\/h2>\n

Azure Cosmos DB is Microsoft’s premier fully managed NoSQL database for modern app development. It is ideal for solutions including artificial intelligence, digital commerce, Internet of Things, booking management and other types of use cases. It offers single-digit millisecond response times, automatic and instant scalability along with guaranteed speed at any scale. Azure Cosmos DB’s NoSQL offering stands out with its exceptional performance and robust feature set that can be an alternative to using graph databases.<\/p>\n

This blog post outlines the approach and best practices, we, the authors learned while working with a customer who wanted to transition a graph workload to Azure Cosmos DB for NoSQL. Our insights are based on our collaboration with various teams, including software engineering and Line-Of-Business (LOB) application teams, at Walmart.<\/p>\n

When should I do this?<\/strong><\/h2>\n

Azure Cosmos DB for NoSQL is recommended for building green-field applications. However, if users are committed to refactoring their code, it can also be used in scenarios where users are attempting to squeeze the most performance for their applications. The scenario highlighted here is an application currently using a graph as the back-end database for storing, querying and analyzing data relationships. In this blog post we will illustrate using the Gremlin graph database and provide guidance to re-model the same into Azure Cosmos DB for NoSQL.<\/p>\n

The three important qualifying factors wherein this approach is valid include:<\/p>\n

    \n
  1. Your use-case leverages point-reads most of the time (> 80%) for querying data.<\/li>\n
  2. Your use-case leverages 1-x traversals most of the time (> 80%) for querying data.<\/li>\n
  3. Your use-case leverages a combination of point-reads and 1-x traversals most of the time (>80%) for querying data.<\/li>\n<\/ol>\n

    This approach is not<\/strong> recommended if you are mostly doing a 2x- and 3x- or more level of traversals with complex edge-relationships and\/or data size of database is > 5 TBs.<\/p>\n

    NOTE: This post is not a recommendation for all customers using Gremlin API or any graph database, to migrate to Azure Cosmos DB for NoSQL. This blog post is intended to highlight how one customer approached this given their needs and the specific factors mentioned above.<\/span><\/div><\/p>\n

    Approach<\/strong><\/h2>\n

    Ideally, the recommended approach is to simulate the Gremlin API data model and vertex (aka node) to vertex traversals via edges using Azure Cosmos DB for NoSQL. The table below documents the source and destination:\u00a0<\/span>\u00a0<\/span><\/p>\n\n\n\n\n\n\n\n
    Currently in Azure Cosmos DB\u00a0<\/strong>for Gremlin<\/strong><\/td>\nProposed in Azure Cosmos DB\u00a0<\/strong>for NoSQL<\/strong><\/td>\n<\/tr>\n
    Vertex (Node)<\/td>\nItem<\/td>\n<\/tr>\n
    Edge (Relationship)<\/td>\nItem<\/td>\n<\/tr>\n
    A forward edge (i.e., a forward traversal). E.g., A –> B<\/td>\nItem<\/td>\n<\/tr>\n
    A reverse edge (i.e., a backward traversal). E.g., B –> A<\/td>\nItem<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    \n<\/span>\nIn essence the approach is to create 4 items.<\/span>\u00a0<\/span><\/p>\n

    Re-modeling \u2013 an example<\/strong><\/h2>\n

    A graph is just a collection of vertices and edges \u2013 or, in other words, a set of nodes and the relationships that connect them. Graphs represent entities as nodes and the ways in which those entities relate to the world as relationships. This general-purpose, expressive structure allows us to model all kinds of scenarios. In the example below, for a retail online sales scenario, we are building a graphical representation of a customer data point. Our vertex has id: Cust001, label: customer and properties including email address, address, first seen, last seen, device1, device2 and so on. Device identified by id Device_1 is in turn shared by customer identified by id: Cust006, and device identified by id Device_2 is shared by two other customers, identified by id: Cust81 and Cust431.<\/span><\/p>\n

    \"Graph<\/a><\/p>\n

    This Gremlin API data model vertex could be represented in <\/span>an <\/span>Azure Cosmos DB for NoSQL data model in the following JSON representation:<\/span>\u00a0<\/span><\/p>\n

    {\r\n\r\n     \"doc_type\": \"node\",\r\n     \"id\": \"<<generate guid>>\",\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\/\/ generate a GUID<\/span>\r\n     \"type\": \"[Type]\",\r\n     \"key\": \"[Key]\",\r\n     \"pk\": \"[Type + Key]\",                      \/\/ generate a partition key<\/span>\r\n     \"first_seen\": \"[first_seen]\",\r\n     \"last_seen\": \"[last_seen]\",\r\n     \"details\": {\r\n              \"a1\": \"v1\",\r\n              \"a2\": \"v2\",\r\n              \".com\": {\r\n                    \"x\": {\r\n                       \"p\": \"abc\",\r\n                       \"g\": \"def\"\r\n                    },\r\n                    \"com_fs\": \"\",\r\n                    \"com_ls\": \"\",\r\n                    \"mp_fs\": \"\",\r\n                    \"mp_ls\": \"\",\r\n                    \"last_\"\r\n     },\r\n}<\/pre>\n
    The <\/span>edge (forward direction) traversal item could be represented as:<\/span><\/span>\u00a0<\/span><\/p>\n
    {\u00a0\r\n     \"doc_type\": \"edge\",\r\n     \"id\": \"<<generate guid>>\",                 \/\/ [Edge.Node1.key].[Edge.Node2.key].[Edge.Type]<\/span>\r\n     \"type\": \"[Type],                           \/\/ alternatively, \"[Edge.Node1.key]\" <\/span>\r\n     \"key\": \"[Key]\",\r\n     \"pk\": \"[Type + Key]\", \r\n     \"first_seen\": \"[first_seen]\",\r\n     \"last_seen\": \"[last_seen]\",\r\n     \"details\": {\r\n              \"a1\": \"v1\",\r\n              \"a2\": \"v2\",\r\n              \".com\": { \r\n                    \"x\": {\r\n                       \"p\": \"abc\",\r\n                       \"g\": \"def\" \r\n                    },\r\n                    \"com_fs\": \"\",\r\n                    \"com_ls\": \"\",\r\n                    \"mp_fs\": \"\",\r\n                    \"mp_ls\": \"\"\r\n                    \"last_\"\r\n     },\r\n}\u00a0\r\n\r\n<\/pre>\n
    A practical example<\/strong><\/h2>\n
    Gremlin API data mode and queries<\/strong><\/h3>\n
    We use the air-routes<\/a> graph model that allows one to plan their air routes in terms of finding the shortest path or multiple routes to reach from a source to a destination. We have also migrated the air-routes data to Azure Cosmos DB for Gremlin which is available here<\/a> for you to experiment with.<\/div>\n
    <\/div>\n
    <\/div>\n
    \"Image<\/a><\/div>\n
    \n
    Click here<\/a> for Image Reference<\/span><\/p>\n
    The GitHub sample contains 4 vertex types. The vertices airport, country<\/span>,<\/span> and continent are important. The version vertex exists for the purpose of tracking the version of the graph. The following shows the number of vertices for each type.<\/span><\/p>\n<\/div>\n
    \"Image<\/a><\/p>\n
    Airport vertex has<\/span> a few<\/span> properties associated with it <\/span>which<\/span> provide<\/span> airport related information like number of runways, elevation<\/span>, geographical location<\/span><\/span>,<\/span><\/span><\/span> etc.<\/span><\/span>\u00a0<\/span><\/p>\n
    \"Image<\/a><\/p>\n
    Routes are edges that connect the countries and the airports<\/span>. They<\/span> have<\/span> a<\/span> distance property to calculate the distance between the two connected airport vertices. <\/span>The f<\/span>ollowing shows the number of edges for each <\/span>type.<\/span><\/span><\/p>\n
    \"Image<\/a><\/p>\n
    The f<\/span>ollowing shows a s<\/span>ample edge connection <\/span>between <\/span>two airport<\/span>s<\/span> with<\/span> properties label and distance between the airports.<\/span><\/p>\n
    \"Image<\/a><\/p>\n
    The <\/span>image below <\/span>represents<\/span> the sample <\/span>mode<\/span>l<\/span> with all the vertices and edges<\/span>.<\/span><\/span><\/p>\n
    \"Image<\/a><\/p>\n
    Let<\/span> us<\/span> execute <\/span>a <\/span>few queries on the model and then simulate the same in <\/span>NoSQL.<\/span><\/span><\/p>\n
    Single Traversal<\/span><\/b><\/h3>\n
    \/\/ List all the airports for the country India\u00a0<\/span>\r\ng.V().has(\"country\",\"desc\",\"India\").out(\"contains\").values(\"city\")<\/span><\/pre>\n
    The output is as exhibited below:<\/span><\/p>\n
    \"Image<\/a>\nApproximate <\/span>Request Unit (<\/span>RU<\/span>)<\/span> consumption = 25 RU\/s<\/strong>\u00a0<\/span><\/span>\u00a0<\/span><\/p>\n
    Double Traversal\u00a0<\/span><\/b>\u00a0<\/span><\/h3>\n
    \/\/ Bangalore to San Francisco with 1 hop\/layover<\/span>\r\n<\/span>g.V().has(\"code\",\"BOM\").out().out().has(\"code\",\"SFO\").path().by(\"code\")<\/span>\u00a0<\/span><\/pre>\n
    The output is as <\/span>exhibited<\/span> below:<\/span><\/p>\n
    \"Image<\/a>\nApproximate <\/span>RU<\/span> consumption = 770 RU\/s<\/strong><\/span><\/span><\/p>\n
    \/\/ Find the top ten overall in terms of incoming routes<\/span>\r\n<\/span>g.V().hasLabel('airport').order().by(__.in('route').count(),decr).limit(10).project('ap','routes').by('code').by(__.in('route').count())<\/span>\u00a0<\/span><\/pre>\n
    Azure Cosmos DB for NoSQL model and queries<\/span><\/b>\u00a0<\/span><\/h2>\n
    Understanding query access patterns is critical before designing a data model for your use-case in Azure Cosmos DB for NoSQL. Based on your application specific query patterns, it is advisable to create a schema and choose the partition key for your container. This results in a highly optimized and cost-efficient data model in Azure Cosmos DB for NoSQL.<\/span>\u00a0<\/span><\/p>\n
    In this case, the schema of the model will include forward and reverse traversal for the vertices and edges. Additionally, the query access pattern will help us in defining the partition key and additional attributes.\u00a0<\/span>\u00a0<\/span><\/p>\n
    A basic Gremlin API data model when translated into a NoSQL data model looks similar to:<\/span><\/p>\n
    \"Image<\/a><\/p>\n
    How do we map vertex and edge to NoSQL documents effectively? <\/span>\u00a0<\/span><\/p>\n
    \"Image<\/a>\nApproximate <\/span>RU<\/span> consumption = <\/span>11,900<\/span> RU\/s<\/span><\/strong><\/span><\/p>\n