{"id":2062,"date":"2022-06-02T10:23:17","date_gmt":"2022-06-02T17:23:17","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/azure-sdk\/?p=2062"},"modified":"2022-06-02T10:23:17","modified_gmt":"2022-06-02T17:23:17","slug":"accelerate-web-forms-using-azure-form-recognizer-client-library","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/azure-sdk\/accelerate-web-forms-using-azure-form-recognizer-client-library\/","title":{"rendered":"Accelerate web forms using Azure Form Recognizer client library"},"content":{"rendered":"<p><em>This post was written by guest blogger and Developer Technologies MVP <a href=\"https:\/\/mvp.microsoft.com\/en-us\/PublicProfile\/33676\">Dino Esposito<\/a>.<\/em><\/p>\n<p>One way in which artificial intelligence (AI) provides concrete help is in the automation of repetitive chores. More generally, AI helps reduce the number of steps required to accomplish a task. Put another way, one of the most relevant aids we can expect from AI in everyday operations is the simplification of workflows. Human operators can achieve the same results with less effort and can avoid error-prone actions.<\/p>\n<p>In this article, I aim at illustrating just one of these scenarios: completing the passport-specific fields of a registration form in some B2C portal. In similar situations, it&#8217;s fairly common today that the backend service requires both:<\/p>\n<ul>\n<li>The actual photo of the passport\u2014typically the first page with picture and details.<\/li>\n<li>Individual items of textual information, such as first and last name, birth date, country of residence, expiry date, and number.<\/li>\n<\/ul>\n<p>Users are often presented with a webpage containing a file upload component and text fields to enter data. The users are expected to manually complete both things. Why not upload the file on its own?<\/p>\n<p>To achieve automatic uploads, there&#8217;s no need for some dedicated OCR service you host on your end. The Azure Form Recognizer client library makes this scenario possible.<\/p>\n<h2>The Form Recognizer Cognitive Service<\/h2>\n<p><a href=\"https:\/\/docs.microsoft.com\/azure\/applied-ai-services\/form-recognizer\/overview\">Azure Form Recognizer<\/a> uses pre-trained machine learning models to extract several types of information from provided documents. Depending upon the class of the provided document, returned information can be in the form of key-value pairs, tables, or plain text. Any returned information is packaged in a JSON container. You need to know what API to call, therefore you must be aware of which document type you&#8217;re submitting. If you don&#8217;t do so, you might still be able to have the document processed but will likely get a low-quality result. It&#8217;s on the roadmap, though, to support document classification. In the future, you can send the document without any classification beforehand. The Azure model will figure out the type, whether a passport or an invoice, and return the appropriate fields. The returned information is often structured enough to be immediately usable as is. In general, Form Recognizer is a powerful tool to automate document data processing in software applications. Usage of the service isn&#8217;t limited to .NET applications.<\/p>\n<p>To start using Form Recognizer from within an application, you first need to create a specific Azure resource. From the Azure home page, you select the Form Recognizer type and confirm to create it. See Figure 1.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-content\/uploads\/sites\/58\/2022\/05\/form-recognizer-create-resource.jpg\" alt=\"Creating a Form Recognizer resource in the Azure portal\" \/><\/p>\n<blockquote><p><strong>Figure 1:<\/strong> Creating a Form Recognizer resource in Azure<\/p><\/blockquote>\n<p>Next, you select an existing subscription, name the new form recognizer and indicate the Azure resource group that will contain it. You also need to specify the Azure region of choice and opt for a pricing tier. A free plan is possible too. The chosen name will determine the actual URL you&#8217;ll be using from within any client application. From the portal, you also download the API key necessary for each and every call.<\/p>\n<h2>Set up a sample ASP.NET Core client application<\/h2>\n<p>For testing the capabilities of the Form Recognizer service, we need to have a host application. For example, an ASP.NET Core application. In this article, I&#8217;ll use the starter template you can freely download from <em>https:\/\/ybq-dev.azurewebsites.net<\/em>. At the end of the day, it&#8217;s a plain ASP.NET Core MVC application with one key controller action and one crucial Razor view. The user interface of the sample page I&#8217;ll discuss is shown in Figure 2.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-content\/uploads\/sites\/58\/2022\/05\/form-recognizer-sample-passport-upload-page.png\" alt=\"The sample passport upload page\" \/><\/p>\n<blockquote><p><strong>Figure 2:<\/strong> The sample passport upload page<\/p><\/blockquote>\n<p>As you can see, the sample page contains a file upload section and a classic array of form text input fields to collect name, country, passport number and relevant dates. The idea is that the user selects the upload button, selects a photo from the local computer, and uploads it to the action ASP.NET controller. As you can see in Figure 3, the photo has been selected but the other text fields are still empty and will be filled on return of the postback action. The postback action takes place via AJAX.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-content\/uploads\/sites\/58\/2022\/05\/form-recognizer-ready-to-upload-passport-photo.png\" alt=\"Ready to upload a passport photo\" \/><\/p>\n<blockquote><p><strong>Figure 3:<\/strong> Ready to upload a passport photo<\/p><\/blockquote>\n<p>The host page has the following markup layout:<\/p>\n<pre><code class=\"language-html\">&lt;form id=\"passport-form\" asp-antiforgery=\"true\" method=\"post\"&gt;\r\n    &lt;hidden id=\"signalrId\" \/&gt;\r\n    &lt;div class=\"row\"&gt;\r\n        &lt;div class=\"col-12 col-md-4 text-center\"&gt;\r\n            &lt;fileupload id=\"passport\"\r\n                        accept=\".jpeg,.jpg,.png\"\r\n                        placeholder=\"&lt;i class='fal fa-passport'&gt;&lt;\/i&gt;\" \/&gt;\r\n            &lt;div class=\"mt-3\"&gt;\r\n                &lt;button disabled id=\"passport-trigger\" \r\n                        type=\"button\" \r\n                        class=\"btn btn-primary\"&gt;UPLOAD&lt;\/button&gt;\r\n            &lt;\/div&gt;\r\n        &lt;\/div&gt;   \r\n        &lt;div&gt;\r\n            &lt;!-- Form fields --&gt;\r\n        &lt;\/div&gt;     \r\n    &lt;\/div&gt;\r\n&lt;\/form&gt;<\/code><\/pre>\n<p>The <code>hidden<\/code> and <code>fileupload<\/code> custom tags are shortcuts for the markup and JavaScript necessary to model a canonical hidden field and a file input tag that&#8217;s smoother to use for the end user. Both tags are coded as ASP.NET Core <a href=\"https:\/\/docs.microsoft.com\/aspnet\/core\/mvc\/views\/tag-helpers\/intro\">Tag Helpers<\/a>, fully integrated in the starter template used for the demo. The hidden field is related to the setup of the SignalR connection that will monitor the server-side interaction with the Azure Form Recognizer resource.<\/p>\n<p>The following code represents the ASP.NET Core controller action method that receives the HTTP POST from the HTML form:<\/p>\n<pre><code class=\"language-csharp\">public class TaskController : Controller\r\n{\r\n    private readonly IHubContext&lt;DefaultMonitorHub&gt; _srContext;\r\n    private readonly TaskService _task;\r\n\r\n    public TaskController(IHubContext&lt;DefaultMonitorHub&gt; srContext) \r\n    {\r\n        _srContext = srContext;\r\n        _task = new TaskService();\r\n    }\r\n\r\n    [HttpPost]\r\n    [ActionName(\"passport\")]\r\n    public async Task&lt;IActionResult&gt; ProcessPassportUpload(\r\n        IFormFile passport,\r\n        string signalrId)\r\n    {\r\n        \/\/ Set up SignalR connection to pass over to the method that\r\n        \/\/ orchestrates the interaction with the recognizer \r\n        var signalR = new ConnectionDescriptor&lt;DefaultMonitorHub&gt;(\r\n            signalrId,\r\n            _srContext);\r\n\r\n        \/\/ Invoke the Form Recognizer client to process the photo\r\n        \/\/ ...\r\n    }\r\n}<\/code><\/pre>\n<p>The file uploaded from the client reaches the controller action method wrapped up in an <code>IFormFile<\/code> object. From here, you open a file stream and trigger the form recognizer. The Form Recognizer client will upload the photo to the Azure service, have it processed, and return the extracted data.<\/p>\n<h2>The Form Recognizer client<\/h2>\n<p>To connect to the previously created Form Recognizer resource from the host ASP.NET Core application, install the <a href=\"https:\/\/www.nuget.org\/packages\/Azure.AI.FormRecognizer\">Azure.AI.FormRecognizer<\/a> NuGet package. The connection point between your client application and the Azure service is the <code>FormRecognizerClient<\/code> class defined in the package. This class lets you arrange a passport analysis in just a few intelligible steps.<\/p>\n<p>Parsing the passport image, therefore, has more of a workflow than of a single operation. Hence, it&#8217;s a good idea to isolate all the steps in an application service method invoked from the controller action method. The <code>_task<\/code> member is defined in the controller&#8217;s constructor.<\/p>\n<pre><code class=\"language-csharp\">\/\/ Invoke the Form Recognizer client to process the photo\r\nvar (mrz, confidence) = await _task\r\n    .ParsePassportFile(passport.OpenReadStream(), signalR);<\/code><\/pre>\n<p>The <code>ParsePassportFile<\/code> method opens the stream out of the uploaded file and starts working with the classes in the Form Recognizer library. A SignalR <code>Hub<\/code> object is also passed to send updates to the browser while the operations take place.<\/p>\n<pre><code class=\"language-csharp\">\/\/ SignalR notification\r\nawait signalR.Hub.Clients.Client(signalR.ConnectionId)\r\n    .SendAsync(\"updateStatus\", \"Uploading file to Azure cloud...\");<\/code><\/pre>\n<p>The client object requires the URL of the dedicated Form Recognizer resource and the related API key.<\/p>\n<pre><code class=\"language-csharp\">\/\/ xxx is the display name of the Azure resource\r\nconst string Endpoint = \"xxx.cognitiveservices.azure.com\";\r\nconst string ApiKey = \"...\";\r\nvar client = new FormRecognizerClient(\r\n    new Uri(Endpoint), \r\n    new AzureKeyCredential(ApiKey));<\/code><\/pre>\n<p>Once you have the client object, method <code>StartRecognizeIdentityDocumentAsync<\/code> takes the stream to the uploaded photo and returns a temporary object of type <code>RecognizeIdentityDocumentsOperation<\/code>. On this object, you call the method <code>WaitForCompletionAsync<\/code>, which covers the server phase that interacts with the trained model.<\/p>\n<pre><code class=\"language-csharp\">var operation = await client.StartRecognizeIdentityDocumentsAsync(stream);\r\nvar response = await operation.WaitForCompletionAsync();<\/code><\/pre>\n<p>The response you get is a collection of recognized forms. In general, the uploaded photo can contain multiple smaller images that can be recognized as known types of forms.<\/p>\n<pre><code class=\"language-csharp\">RecognizedFormCollection identityDocuments = response.Value;\r\nif (identityDocuments.Count == 0)\r\n    return (null, 0);\r\n\r\nRecognizedForm identityDocument = identityDocuments.FirstOrDefault();\r\nif (identityDocument == null)\r\n    return (null, 0);<\/code><\/pre>\n<p>In this example, I focus on the first returned document. In a realistic scenario, you might want to apply any selection logic that fits. For example, processing all returned documents or only documents with a given number of pages or of a certain type. Here&#8217;s a list of properties you can query on the <code>RecognizedForm<\/code> class.<\/p>\n<table>\n<thead>\n<tr>\n<th>Property<\/th>\n<th>Description<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>Fields<\/code><\/td>\n<td>List of information items recognized in the document, such as the MRZ area in a passport<\/td>\n<\/tr>\n<tr>\n<td><code>FormType<\/code><\/td>\n<td>Type of the document. For example, Invoice or Passport.<\/td>\n<\/tr>\n<tr>\n<td><code>FormTypeConfidence<\/code><\/td>\n<td>Value between 0 and 1 to denote the system&#8217;s confidence in the response<\/td>\n<\/tr>\n<tr>\n<td><code>ModelId<\/code><\/td>\n<td>Further name (if any) of the recognized document<\/td>\n<\/tr>\n<tr>\n<td><code>PageRange<\/code><\/td>\n<td>Pages of the document<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>The actual content recognized in the scanned document is returned as a collection of fields. The next step, therefore, is just getting ahold of the list of fields for the recognized type of document and extract information. Here&#8217;s the information captured when a valid passport first page photo is uploaded.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-content\/uploads\/sites\/58\/2022\/05\/form-recognizer-info-captured.png\" alt=\"Information captured when a valid passport first page photo is uploaded\" \/><\/p>\n<p>The collection of fields contains just one element, the document has only one page, and the type is passport. The list of possible fields is outlined in the service documentation at <a href=\"https:\/\/docs.microsoft.com\/azure\/applied-ai-services\/form-recognizer\/concept-id-document\">Form Recognizer ID document model<\/a>.<\/p>\n<p>For a passport, the sole field returned is <code>MachineReadableZone<\/code> (MRZ) which corresponds to the two (or maybe three) lines of text at the bottom of the page interspersed with several angle bracket characters. The following code returns the passport MRZ as a single continuous string with a blank in lieu of line breaks.<\/p>\n<pre><code class=\"language-csharp\">if (!identityDocument.Fields.TryGetValue(\"MachineReadableZone\", out FormField mrzField))\r\n    return (null, 0);\r\nvar mrz = mrzField.ValueData.Text;<\/code><\/pre>\n<p>As mentioned, the MRZ string you obtain in this way contains some blanks. With a call to <code>String.Replace<\/code>, you remove them and obtain a continuous string as if the passport MRZ rows were just one.<\/p>\n<pre><code class=\"language-csharp\">mrz = mrz.Replace(\" \", \"\");<\/code><\/pre>\n<p>The final step consists in parsing the MRZ record to extract first and last name, date of birth, passport number, gender, expiry, and issuing country.<\/p>\n<h2>Parse the MRZ record<\/h2>\n<p>The MRZ record follows an international format made of plain information laid out at specific lengths and many check digits. The sample application provides an MRZ parser class articulated in two steps. First, it extracts strings following the known schema and computes and compares check digits. Second, it cleans the data and, for example, it converts date strings in plain <code>DateTime<\/code> .NET objects. As a result, one gets a <code>PassportData<\/code> C# data structure that contains all relevant passport information. The data structure, along with the value of model confidence, is serialized to JSON and returned to the caller. The following snippet shows the final lines of code in the controller action method.<\/p>\n<p>In version 2.1 of the client library, you also find some automatic parsing of MRZ. In version 3.0, instead, you find more fields, improved model quality, and automatic recognition of document type.<\/p>\n<pre><code class=\"language-csharp\">var (mrz, confidence) = await _task\r\n    .ParsePassportFile(passport.OpenReadStream(), signalR);\r\nreturn mrz == null \r\n    ? Json(new {passport = new PassportData(false), confidence = confidence})\r\n    : Json(new {passport = mrz.Data, confidence = confidence});<\/code><\/pre>\n<p>To finish, let&#8217;s see how the downloaded information makes its way back to the origin HTML page.<\/p>\n<h2>Update the requesting webpage<\/h2>\n<p>The call to Form Recognizer starts from an HTML button. There, it returns JSON data ready to be incorporated in the live page. Here&#8217;s an example of the JavaScript code responsible for parsing the JSON response:<\/p>\n<pre><code class=\"language-javascript\">function (data) {\r\n    var response = JSON.parse(data);\r\n    if (response.passport.valid) {\r\n        console.log(response.passport.firstName);\r\n        console.log(response.passport.lastName);\r\n        console.log(response.passport.issuerCountryCode);\r\n        console.log(response.passport.dateOfExpiry.substring(0, 10));\r\n        console.log(response.passport.dateOfBirth.substring(0, 10));\r\n        console.log(response.passport.number);\r\n    }\r\n}<\/code><\/pre>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-content\/uploads\/sites\/58\/2022\/05\/form-recognizer-sample-page-after.png\" alt=\"The sample page after Form Recognizer\" \/><\/p>\n<blockquote><p><strong>Figure 5:<\/strong> The sample page after the form recognizer<\/p><\/blockquote>\n<p>In summary, the simple upload of a passport photo triggered a call to the Azure Form Recognizer service and returned a parsed passport class to fill out remaining input fields. This feature saves the end user from some mundane data entry. It doesn&#8217;t cost much to arrange. Furthermore, it smooths the user experience in web forms that require uploading passport information, or form data in general.<\/p>\n<h2>What&#8217;s new in library version 3.0<\/h2>\n<p>The code shown above refers to version 2.1 of the library. In the 3.0 version, new classes were added. And as far as ID documents are concerned, the work of parsing the MRZ string is done automatically by the API. Here&#8217;s an example that gets the last name of the passport holder directly. To use this new API, you just need to upgrade the related NuGet package.<\/p>\n<pre><code class=\"language-csharp\">\/\/ Using version 3.0\r\nvar client = new DocumentAnalysisClient(\r\n    new Uri(Endpoint),\r\n    new AzureKeyCredential(ApiKey));\r\nvar operation = \r\n    await client.StartAnalyzeDocumentAsync(\"prebuilt-idDocument\", stream);\r\nawait operation.WaitForCompletionAsync();\r\nAnalyzeResult result = operation.Value;\r\nAnalyzedDocument doc = result.Documents.FirstOrDefault();\r\n\r\nif (doc == null)\r\n    return (null, 0);\r\n\r\n\/\/ Extract MRZ info\r\nif (!doc.Fields.TryGetValue(\"MachineReadableZone\", out DocumentField mrzField))\r\n    return (null, 0);\r\n\r\n\/\/ Extract last name\r\nvar mrzParts = mrzField.AsDictionary();\r\nvar ln = mrzParts[\"LastName\"].Content;<\/code><\/pre>\n<p>The Form Recognizer service is continually updated. Information about the features of the latest version is available at https:\/\/docs.microsoft.com\/azure\/applied-ai-services\/form-recognizer\/whats-new?tabs=csharp.<\/p>\n<h2>More about the Form Recognizer service<\/h2>\n<p>The Form Recognizer service&#8217;s free tier:<\/p>\n<ul>\n<li>Works with images up to 4 MB\u2014no smaller than 50&#215;50 pixels and no larger than 10000&#215;10000.<\/li>\n<li>Allows up to 500 pages per month at a maximum rate of 20 calls-per-minute.<\/li>\n<\/ul>\n<p>The pre-trained model works sufficiently for standard passports without requiring further training. Finally, in addition to the REST API, the Form Recognizer service is also available through a Docker container to run the AI engine in your environment.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Learn how to use the Azure Form Recognizer client library for .NET to populate a registration form.<\/p>\n","protected":false},"author":89261,"featured_media":2069,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[701,706,767],"class_list":["post-2062","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure-sdk","tag-net","tag-azuresdk","tag-cognitive"],"acf":[],"blog_post_summary":"<p>Learn how to use the Azure Form Recognizer client library for .NET to populate a registration form.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts\/2062","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/users\/89261"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/comments?post=2062"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts\/2062\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/media\/2069"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/media?parent=2062"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/categories?post=2062"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/tags?post=2062"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}