{"id":4575,"date":"2017-09-28T10:02:12","date_gmt":"2017-09-28T17:02:12","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/?p=4575"},"modified":"2020-03-14T19:59:34","modified_gmt":"2020-03-15T02:59:34","slug":"data-independent-graphql-using-view-model-based-schemas","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/data-independent-graphql-using-view-model-based-schemas\/","title":{"rendered":"Creating a Data Visualization GraphQL Server with a Loosely Coupled Schema"},"content":{"rendered":"<p>We recently <a href=\"https:\/\/www.microsoft.com\/developerblog\/2017\/09\/26\/custom-analytics-dashboard-application-insights\/\">built a dashboard for visualizing data<\/a> from\u00a0<a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/application-insights\/\">Application Insights<\/a>. This Azure service enables the collection of large amounts of information about an application&#8217;s business analytics and health, plus quick and efficient access to that data by using <a href=\"https:\/\/kusto.azurewebsites.net\/docs\/\">Kusto Query Language<\/a>. For this purpose of building a flexible data visualization dashboard, we explored the <a href=\"http:\/\/graphql.org\/\">GraphQL<\/a> stack, a query language that enables abstraction of server-side API calls under a single neat wrapper instead of multiple endpoints.<\/p>\n<p>The initial design of <a href=\"https:\/\/github.com\/CatalystCode\/ibex-dashboard\">Project Ibex<\/a>, our <a href=\"https:\/\/docs.microsoft.com\/en-us\/bot-framework\/portal-analytics-overview\">Bot Analytics<\/a> dashboard,\u00a0included a centralized data pipeline using a REST server as a pass-through proxy to return aggregated results from multiple services with a single JSON response. Maintaining our own data service, however, added unnecessary complexity to the Ibex project. Giving the rising popularity of GraphQL, we decided to investigate a design based on this stack. We sought to minimize the existing complexity and optionally gain performance improvement.<\/p>\n<p>One of the most powerful features of Ibex is the ability for users to specify any Application Insights query when building a new dashboard, without writing a single line of code. However, to use GraphQL, users must define a common schema between their clients and server. Since\u00a0GraphQL requires the client to have access to a defined schema in advance, there may be some considerations when developing with it: for instance, what happens when a project like Ibex needs to support dynamic data representations?<\/p>\n<p>At first glance, it seems as there is no simple solution to support <em>any<\/em> data and <em>any<\/em> client-provided query while keeping the schema loosely coupled with the data representation. In this blog post, we will share our learnings from this challenge, including how we leveraged the GraphQL stack and came up with a smart yet simple solution.<\/p>\n<p><!--more--><\/p>\n<h2>Rethinking GraphQL schemas<\/h2>\n<p><figure id=\"attachment_4845\" aria-labelledby=\"figcaption_attachment_4845\" class=\"wp-caption aligncenter\" ><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2017\/09\/bot-framework-preview.png\" alt=\"Image bot framework preview\" width=\"1522\" height=\"721\" class=\"aligncenter size-full wp-image-10889\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/09\/bot-framework-preview.png 1522w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/09\/bot-framework-preview-300x142.png 300w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/09\/bot-framework-preview-1024x485.png 1024w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/09\/bot-framework-preview-768x364.png 768w\" sizes=\"(max-width: 1522px) 100vw, 1522px\" \/><figcaption id=\"figcaption_attachment_4845\" class=\"wp-caption-text\">Project Ibex &#8211; Screenshot<\/figcaption><\/figure><\/p>\n<p>While we could have worked around GraphQL&#8217;s dynamic schema limitations by supporting a predefined set of queries, this approach would have limited the reusability of Ibex and increased long-term maintenance complexity.<\/p>\n<p>Another workaround would have been to declare a generic schema GraphQL, as follows:<\/p>\n<pre class=\"lang:js decode:true\">\/\/ #### NOT recommended approach ####\r\n\r\ntype Query {\r\n  generalQuery(query: String!): JsonResponse\r\n}\r\n\r\ntype JsonResponse {\r\n  body: String\r\n}\r\n<\/pre>\n<p>The results contain a single field <em>&#8216;JsonResponse&#8217;<\/em> and it is the client&#8217;s responsibility to parse, process and visualize it. The above solution, enables dynamic schema support at the expense of GraphQL benefits, reducing GraphQL into an expensive REST server with no additional benefits.<\/p>\n<p>With a generalized schema:<\/p>\n<ul>\n<li>Clients are unable to specify which data they want to retrieve<\/li>\n<li>Clients suffer degraded performance as a result of redundant data transfer<\/li>\n<\/ul>\n<p>The role of Ibex&#8217;s server is to provide data for <em>dashboards. <\/em>The set of UI components that a dashboard can show is naturally limited and very small (line charts, bar charts, gauge charts, etc.). Rethinking and redefining the schema based on this view model rather than on the data itself is key to this challenge.<\/p>\n<h3>GraphQL Server as a View Model<\/h3>\n<p>We learned through investigation that it is possible to define a known client schema around a Data-View model to enable support for data\u00a0independent dynamic queries with GraphQL.<\/p>\n<p>For specific cases when the schema is dynamically changing between calls, or unknown in advance, but at the same time the client is naturally limited, we suggest a <em><strong>view model-based<\/strong> <\/em><strong>schema<\/strong>. Instead of defining it based on the data, we can build it based on the consumers, and the UI components. Since we know in advance which UI components are supported, we can preprocess the data on the server and return it to the format supported by the UI components.<\/p>\n<p>Consider the following example, Imagine we want to present a pie chart that shows how many requests to a bot each user made. One option would be to retrieve all requests for all the users, sum up the data by user and visualize it. To make this option work, the GraphQL schema should be defined based on the Application Insights requests table&#8217;s schema.<\/p>\n<pre class=\"lang:default decode:true \">type Query {\r\n  requests(query: String!, appId:String!, apiKey:String!): [Request]\r\n}\r\n\r\ntype Request{\r\n  timestamp: [String]\r\n  id: [String]\r\n  url: [String]\r\n  resultCode: [Int]\r\n  client_IP: [String]\r\n  ...\r\n}<\/pre>\n<p>This option will result in raw data and the schema is strongly coupled with the data representation. Any change to the data or query should result in a schema change.<\/p>\n<p>A second option would be to return the summarized data:<\/p>\n<pre class=\"lang:default highlight:0 decode:true\">{\r\n  \"data\": {\r\n    ],\r\n    \"pieCharts\": [\r\n      {\r\n        \"labels\": [\r\n          \"Jon Doe &lt;19.23.4.244&gt;\",\r\n          \"Bob &lt;12.25.200.255&gt;\"\r\n        ],\r\n        \"values\": [\r\n          2,\r\n          1\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}<\/pre>\n<p> <img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2017\/09\/PieExample.png\" alt=\"Image PieExample\" width=\"220\" height=\"119\" class=\"aligncenter size-full wp-image-10892\" \/><\/p>\n<p>When dealing with a pie chart, we expect the data to be in the form of [ {label<sub>1<\/sub>, value<sub>1<\/sub>} , {label<sub>2<\/sub>, value<sub>2<\/sub>}, &#8230;, {label<sub>N<\/sub>, value<sub>N<\/sub>}]. Every other information is <strong>redundant<\/strong>. The same goes for other UI components: a line chart graph would contain several <em>&#8216;line series&#8217;<\/em> and a bar chart would consist of<em> &lt;label, value&gt;<\/em> pairs.<\/p>\n<p>The resulting schema would look like:<\/p>\n<pre class=\"lang:default decode:true\">type Query {\r\n  pieCharts(query: String!, appId:String!, apiKey:String!): [PieChart]\r\n  lineCharts(query: String!, appId:String!, apiKey:String!): [LineChart]\r\n  barCharts(query: String!, appId:String!, apiKey:String!): [BarChart]\r\n  ...\r\n}\r\n\r\ntype PieChart {\r\n  labels: [String]\r\n  values: [Int]\r\n}\r\n\r\ntype LineChart {\r\n  id: String\r\n  seriesData : [Series]\r\n}\r\n\r\ntype BarChart {\r\n  id: String\r\n  seriesData : [Series]\r\n}\r\n\r\ntype Series {\r\n  label: String\r\n  x_values: [String]\r\n  y_values: [Int]\r\n}<\/pre>\n<p>To bind the actual data to the schema, a <em>resolver<\/em> method must be defined; the resolver fetches the data. The resolver method might use REST API to call the actual data source (e.g., Application Insights). The resulting JSON would then pass through a &#8216;translation layer&#8217; to transform the raw JSON into a new JSON response, <strong><em>in the format expected by the UI component<\/em><\/strong>.<\/p>\n<p><figure id=\"attachment_4852\" aria-labelledby=\"figcaption_attachment_4852\" class=\"wp-caption aligncenter\" ><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2017\/09\/diagram.png\" alt=\"Image diagram\" width=\"945\" height=\"557\" class=\"aligncenter size-full wp-image-10890\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/09\/diagram.png 945w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/09\/diagram-300x177.png 300w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/09\/diagram-768x453.png 768w\" sizes=\"(max-width: 945px) 100vw, 945px\" \/><figcaption id=\"figcaption_attachment_4852\" class=\"wp-caption-text\">High-level sequence diagram<\/figcaption><\/figure><\/p>\n<p>Imagine that we want to generate three charts instead of just one: a line chart, a pie chart, and a bar chart.<\/p>\n<p>With GraphQL a user can create a simple data independent query as follows:<\/p>\n<pre class=\"lang:default decode:true\">{\r\n  lineCharts(query: \"some free text query for line charts\") {\r\n    seriesData {\r\n      label\r\n      x_values\r\n      y_values\r\n    }\r\n  }\r\n\r\n  pieCharts(query: \"some free text query for pie charts\") {\r\n    labels\r\n    values\r\n  }\r\n\r\n  barCharts(query: \"some free text query for bar charts\") {\r\n    seriesData {\r\n      label\r\n      x_values\r\n      y_values\r\n    }\r\n  }\r\n}\r\n<\/pre>\n<p>The result of this query would be:<\/p>\n<pre class=\"lang:default decode:true \">{\r\n  \"data\": {\r\n    \"lineCharts\": [\r\n      {\r\n        \"seriesData\": [\r\n          {\r\n            \"label\": \"webchat\",\r\n            \"x_values\": [\r\n              \"2017-05-14T09:35:00Z\", ..., \"2017-07-10T09:10:00Z\"\r\n            ],\r\n            \"y_values\": [\r\n              1, ..., 3\r\n            ]\r\n          },\r\n          {\r\n            \"label\": \"skype\",\r\n            \"x_values\": [\r\n              \"2017-05-15T10:30:00Z\", ..., \"2017-07-10T09:10:00Z\"\r\n            ],\r\n            \"y_values\": [\r\n              2, ..., 1\r\n            ]\r\n          }\r\n        ]\r\n      }\r\n    ],\r\n    \"pieCharts\": [\r\n      {\r\n        \"labels\": [\r\n          \"Number_Of_Inactive_Users\", \"Number_Of_Active_Users\"\r\n        ],\r\n        \"values\": [\r\n          19, 3\r\n        ]\r\n      }\r\n    ],\r\n    \"barCharts\": [\r\n      {\r\n        \"seriesData\": [\r\n          {\r\n            \"label\": \"slack\",\r\n            \"x_values\": [\r\n              \"San Jose\", ..., \"Washington\"\r\n            ],\r\n            \"y_values\": [\r\n              85, ..., 67\r\n            ]\r\n          },\r\n          {\r\n            \"label\": \"skype\",\r\n            \"x_values\": [\r\n              \"Dublin\", ..., \"Wilmington\"\r\n            ],\r\n            \"y_values\": [\r\n              25, ..., 32\r\n            ]\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}<\/pre>\n<p>Notice that the data returned from the server is already processed; it is in the format expected by the UI components and is ready to be consumed. By transforming our data-driven schema into a view model-driven schema, we can now support general queries. All the client needs to do is make sure the query&#8217;s output maps to fit the transformation layer.<\/p>\n<h2>Code<\/h2>\n<p>You can find the <a href=\"https:\/\/github.com\/CatalystCode\/ibex-dashboard-apollo-graphql\/\">source code for the Ibex on GraphQL prototype<\/a> on GitHub. Feel free to contribute.<\/p>\n<p>We have decided to keep this version as a <em>prototype<\/em> because limitations such as lack of &#8216;forks&#8217; (Application Insights queries aggregation) in GraphQL would have resulted in exceeding the queries limit.<\/p>\n<h2>Reuse Opportunities<\/h2>\n<p>Since our server still needs to pull information from some REST endpoints, such as Application Insights, GraphQL does not eliminate all redundant data transfer. However, for data endpoints that are local or support partial data queries, migrating to GraphQL eliminates all redundant data transfer.<\/p>\n<p>For projects with <em>similar requirements<\/em>, where the data&#8217;s structure is not known in advance and the client is limited by the number of supported query types, the server should remain as flexible as possible. To ensure server flexibility it is important to rethink the conventional schema modeling paradigm. By modeling a schema around a view model, it is possible to create data-independent GraphQL implementations.<\/p>\n<hr \/>\n<p><em>Header\u00a0Image: <a href=\"https:\/\/commons.wikimedia.org\/wiki\/File:Social_Network_Analysis_Visualization.png\">Social Network Analysis Visualization<\/a> by <a href=\"https:\/\/commons.wikimedia.org\/wiki\/User:SlvrKy\">Martin Grandjean<\/a>, used under\u00a0<a class=\"extiw\" title=\"w:en:Creative Commons\" href=\"https:\/\/en.wikipedia.org\/wiki\/en:Creative_Commons\">Creative Commons<\/a> <a class=\"external text\" href=\"https:\/\/creativecommons.org\/licenses\/by-sa\/3.0\/deed.en\" rel=\"nofollow\">Attribution-Share Alike 3.0 Unported<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We share how we built a flexible data visualization dashboard by leveraging the GraphQL stack.<\/p>\n","protected":false},"author":21395,"featured_media":10891,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[13,17],"tags":[64,190,249,307],"class_list":["post-4575","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bots","category-frameworks","tag-azure-application-insights","tag-graphql","tag-microsoft-bot-framework-mbf","tag-reactjs"],"acf":[],"blog_post_summary":"<p>We share how we built a flexible data visualization dashboard by leveraging the GraphQL stack.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/4575","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/21395"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=4575"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/4575\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/10891"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=4575"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=4575"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=4575"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}