{"id":8487,"date":"2018-05-07T10:20:03","date_gmt":"2018-05-07T17:20:03","guid":{"rendered":"https:\/\/www.microsoft.com\/developerblog\/?p=8487"},"modified":"2020-03-19T13:38:52","modified_gmt":"2020-03-19T20:38:52","slug":"handwriting-detection-and-recognition-in-scanned-documents-using-azure-ml-package-computer-vision-azure-cognitive-services-ocr","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/handwriting-detection-and-recognition-in-scanned-documents-using-azure-ml-package-computer-vision-azure-cognitive-services-ocr\/","title":{"rendered":"Making sense of Handwritten Sections in Scanned Documents using the Azure ML Package for Computer Vision and Azure Cognitive Services"},"content":{"rendered":"

Business Problem<\/strong><\/h2>\n

For businesses of all sorts, one of the great advantages of the shift from physical to digital documents is the fast and effective search and knowledge extraction methods now available. Gone are the days of reviewing documents line-by-line to find particular information. However, things get more complicated when the researcher needs to extract general concepts, rather than specific phrases. And it\u2019s even more complicated when applied to mixed-quality scanned documents containing handwritten annotations.<\/p>\n

Microsoft recently teamed with EY (Ernst & Young Global Limited) to improve its contract search and knowledge extraction results.\u00a0 EY\u2019s professional services personnel spend significant amounts of time reviewing clients\u2019 contracts in order to extract information about relevant concepts of interest. Automated entity and knowledge extraction from these contracts would significantly reduce the amount of time their staff need to spend on the more mundane elements of this review work.<\/p>\n

It is challenging to achieve acceptable extraction accuracy when applying traditional search and knowledge extraction methods to these documents. Chief among these challenges are poor document image quality and handwritten annotations. The poor image quality stems from the fact that these documents are frequently scanned copies of signed agreements, stored as PDFs, often one or two generations removed from the original. This causes many optical character recognition (OCR) errors that introduce nonsense words. Also, most of these contracts include handwritten annotations which amend or define critical terms of the agreement. The handwriting legibility, style, and orientation varies widely; and the handwriting can appear in any location on the machine-printed contract page. Handwritten pointers and underscoring often note where the handwriting should be incorporated into the rest of the printed text of the agreement.<\/p>\n

We collaborated with EY to tackle these challenges as part of their search and knowledge extraction pipeline.<\/p>\n

Technical Problem Statement<\/h2>\n

Despite recent progress, standard OCR technology performs poorly at recognizing handwritten characters on a machine-printed page. The recognition accuracy varies widely for the reasons described above, and the software often misplaces the location of the handwritten information when melding it in line with the adjoining text. While pure handwriting recognizers have long had stand-alone applications, there are few solutions that work well with document OCR and search pipelines.<\/p>\n

In order to enable entity and knowledge extraction from documents with handwritten annotations, the aim of our solution was first to identify handwritten words on a printed page, then recognize the characters to transcribe the text, and finally to reinsert these recognized characters back into the OCR result at the correct location. For a good user experience, all this would need to be seamlessly integrated into the document ingestion workflow.<\/p>\n

Approach<\/h2>\n

In recent years, computer vision object detection<\/a> models using deep neural networks have proven to be effective at a wide variety<\/a> of object recognition tasks, but require a vast amount of expertly labeled training data. Fortunately, models pre-trained on standard datasets such as COCO<\/a>, containing millions of labeled images, can be used to create powerful custom detectors with limited data via transfer learning<\/a> \u2013 a method of fine-tuning an existing model to accomplish a different but related task. Transfer learning has been demonstrated to dramatically reduce the amount of training data required to achieve state-of-the-art accuracy for a wide range of applications<\/a>.<\/p>\n

For this particular case, transfer learning from a pre-trained model was an obvious choice, given our small sample of labeled handwritten annotation and the availability of relevant state-of-the-art pre-trained models.<\/p>\n

Our workflow, from <\/span>object detection to handwriting recognition and replacement in the contract image OCR result, is summarized in Figure 1 below. To start, we applied a custom object detection model on an image of a contract printed page to detect handwriting and identify its bounding box.<\/span><\/p>\n

The sample Jupyter notebook for object detection and customizable utilities and functions for data preparation and transfer learning in the new Azure ML Package for Computer Vision<\/a> (AML-PCV) made our work much easier.\u00a0The AML-PCV notebook and supporting utilities take advantage of the Faster R-CNN<\/a> object detection model with Tensorflow back-end, which has produced state-of-the-art results in object detection challenges in the field.<\/p>\n

Our project Jupyter Notebooks<\/a> using AML-PCV\u00a0are available on our project GitHub repo<\/a>.\u00a0 You can find more details on the implementation and customizable parameters in AML-PCV available on the Tensorflow object detection website<\/a>. AML-PCV comes with support for transfer learning using faster_rcnn_resnet50_coco_2018_01_28, a model trained on the Coco Common Object in Context dataset<\/a> containing more than 200k labeled images and 1.5 million object instances across 80 categories.<\/p>\n

For our custom application, we used the Visual Object Tagging Tool<\/a> (VOTT) to manually label a small set of public government contract data<\/a>\u00a0containing both machine-printed text and handwriting, as we\u2019ll detail in the data section below. We labelled two classes of handwriting objects in the VOTT tool \u2013 signatures and non-signature (general text such as dates) \u2013 recording the bounding box and label for each instance. This set of labeled data were passed into the AML-PCV notebook to train a custom handwriting detection model.<\/p>\n

Once we had recognized the handwritten annotations, we used the Microsoft Cognitive Services Computer Vision API<\/a> to apply OCR to recognize the characters of the handwriting. You can find the Jupyter Notebooks for this project, and a sample of the data on the project GitHub repo<\/a>.<\/p>\n

\"Image

Figure 1. Model Workflow<\/figcaption><\/figure><\/p>\n

The Data<\/h2>\n

Using VOTT allowed us to produce a training set of 182 labelled images from a sample of Government contracts<\/a> in a matter of a few hours. We drew our test set from an additional 100 contract images, chosen from different states than the training set. As described in the approach, we labelled two classes: handwritten signatures and handwritten non-signatures. Our objective was primarily to correctly interpret the non-signature objects, as these were germane to the entities and concepts we were trying to extract. The signatures typically did not contain this payload. Classifying signature handwriting as a different class allowed us focus on the non-signature handwriting that was of interest.<\/p>\n

The output of VOTT writes an XML file for each image in Pascal-VOC format, with bounding box location information for each labelled object. This format can be read into the AML-PCV directly, with further processing done by utilities called from the notebook. You can access the full set of images and labeled data from this project on an Azure blob public data repository with URI https:\/\/handwriting.blob.core.windows.net\/leasedata. You can also find a smaller sample of the data in the project GitHub repo<\/a>. Figure 2 shows an example of a typical contract section with relevant handwritten parts \u2013 in this case the start date of a real estate lease.<\/p>\n

\"Image

Figure 2. Screenshot of a Contract with Handwriting<\/figcaption><\/figure><\/p>\n

Method<\/h2>\n

Here we provide detail on using the vision toolkit to train the custom object detection model.<\/p>\n

One of the key timesavers provided by the AML-PCV utilities\u00a0is the utilities to recognize, format, and pre-process our labeled training and test data. The code below will import the VOTT labeled dataset and pre-process images to create a suitable training set for the Faster-RCNN model:<\/p>\n

import os, time\r\nfrom cvtk.core import Context, ObjectDetectionDataset, TFFasterRCNN\r\nfrom cvtk.utils import detection_utils\r\n\r\nimage_folder = \"<input image folder including subfolders of jpg and xml>\" # training data from VOTT labeling tool\r\nmodel_dir = \"<saved model directory>\" # dir for saved training models\r\nimage_path = \"<test image path>\" # scoring image path\r\nresult_path = \"<results path>\" # dir for saving images with detection boxes and placeholder text\r\ndata_train = ObjectDetectionDataset.create_from_dir_pascal_voc(dataset_name='training_dataset', data_dir=image_folder)\r\n<\/pre>\n
Below are the default hyperparameter selections for our handwriting object detection model. Since the default minibatch size is set to 1, we set the num_steps equal to the num_epochs multiplied by the number of images in the training set. The learning rate and step number are default parameters from AML-PCV and discussed in details in the notebook<\/a>.<\/p>\n
score_threshold = 0.0       # Threshold on the detection score, use to discard lower-confidence detections.\r\nmax_total_detections = 300  # Maximum number of detections. A high value will slow down training but might increase accuracy.\r\nmy_detector = TFFasterRCNN(labels=data_train.labels, \r\n                           score_threshold=score_threshold, \r\n                           max_total_detections=max_total_detections)\r\n# to get good results, use a larger value for num_steps, e.g., 5000.\r\nnum_steps = len(dataset_train.images)*30\r\nlearning_rate = 0.001 # learning rate\r\nstep1 = 200 \r\n\r\nstart_train = time.time()\r\nmy_detector.train(dataset=data_train, num_steps=num_steps, \r\n                  initial_learning_rate=learning_rate,\r\n                  step1=step1,\r\n                  learning_rate1=learning_rate)\r\nend_train = time.time()\r\nprint(\"the total training time is {}\".format(end_train-start_train))\r\n<\/pre>\n
With these parameter settings and our training set of 182 images, training took 4080 seconds on a standard Azure NC6 DLVM (Azure Deep Learning Virtual Machine<\/a>), with one GPU.<\/p>\n
Let\u2019s examine the pipeline on one government contract page below. Figure 3 shows the detected handwriting boxes in the contract \u2013 blue for non-signature and green for signature. If we are able to recognize the handwritten words, the challenge then is to decide where to insert this in the output of the printed text OCR process.<\/p>\n
Our approach was to generate unique tokens designed to be reliably transcribed by the OCR software. We then inserted those tokens in the areas where we detected handwritten text,\u00a0replacing the original handwritten section, and using these tokens as anchor points in the OCR output. After some experimentation, we proceeded with tokens comprising 5 digits, starting and ending with the number \u201c8\u201d, with 3 randomly generated numbers in between. Figure 4 shows the result of replacing the original text with these numbers.<\/p>\n
\"Image

Figure 3. Annotated Handwriting in One Page of PDF Contract<\/figcaption><\/figure><\/p>\n
 <\/p>\n
\"Image
Figure 4. Inserted Placeholder Texts in Each Detected Handwriting Box<\/figcaption><\/figure><\/p>\n
 <\/p>\n
We then used the Microsoft Cognitive Services Computer Vision API<\/a> OCR service to transcribe each detected handwriting box. Below is a helper function from our notebook to call to the Computer Vision API and return recognized characters.<\/p>\n
def pass2CV(img_array):\r\n    import requests\r\n    \r\n    print ('pass image to computer vision API ......')\r\n    image_data = cv2.imencode('.jpg', img_array)[1].tostring() # convert image array to image bytes\r\n    \r\n    vision_base_url = \"https:\/\/westus2.api.cognitive.microsoft.com\/vision\/v1.0\/\"  \r\n    #International URLs: Replace 'westus2' with your local area, for example 'westeurope' or 'eastasia\u2019.\r\n    #Full list of internal URL\u2019s is available at this link:  \r\n    #https:\/\/westus.dev.cognitive.microsoft.com\/docs\/services\/56f91f2d778daf23d8ec6739\/operations\/56f91f2e778daf14a499e1fa\r\n\r\n    text_recognition_url = vision_base_url + \"RecognizeText\"\r\n\r\n    # change to your Computer Vision API subscription key\r\n    headers  = {'Ocp-Apim-Subscription-Key': subscription_key,\r\n               \"Content-Type\": \"application\/octet-stream\"}\r\n    params   = {'handwriting' : True}\r\n\r\n    response = requests.post(text_recognition_url, headers=headers, params=params, data=image_data)\r\n    response.raise_for_status()\r\n\r\n    operation_url = response.headers[\"Operation-Location\"]\r\n\r\n    import time\r\n\r\n    analysis = {}\r\n    while not \"recognitionResult\" in analysis:\r\n        response_final = requests.get(response.headers[\"Operation-Location\"], headers=headers)\r\n        analysis       = response_final.json()\r\n        time.sleep(1)\r\n\r\n    polygons = [(line[\"boundingBox\"], line[\"text\"]) for line in analysis[\"recognitionResult\"][\"lines\"]]\r\n    \r\n    return polygons\r\n<\/pre>\n
Meanwhile, we built a dictionary matching the anchor digits with the handwriting OCR results. This simplifies our final task of replacing the anchor strings in the printed OCR results. Below, we see a sample of the dictionary for handwriting boxes in Figure 3.<\/p>\n
{'81548': 'couken Chapman 1 ( signature )',\r\n '83428': 'Mhub',\r\n '83728': 'U a L ( Signature )',\r\n '87018': 'ton w 13901',\r\n '87598': 'EO',\r\n '88078': '12 2 State St , BIX ( Addr',\r\n '88298': '2',\r\n '88488': 'I ( Signature ) 2 th'}\r\n<\/pre>\n
In the last step, we matched the handwriting detected blocks with the dictionary and replaced the placeholder text with the OCR result from the Computer Vision API. Figure 5a shows OCR results for the contract page, where the placeholder text is detected well. Then, based on the dictionary above, we replaced the digits with the handwriting OCR results from Computer Vision API. The results are shown in Figure 5b.<\/p>\n
<\/p>\n
\"Figure
Figure 5a. CV API Results of One Page of Contract with Placeholder Text<\/figcaption><\/figure><\/p>\n
 <\/p>\n
\"Image
Figure 5b. Match the Detected Handwriting Back into the Contract Text<\/figcaption><\/figure><\/p>\n
Results<\/h2>\n
There were two main components to this project: handwriting object detection and handwriting OCR. The results on detecting handwritten words were promising. Transcribing the handwritten text was less successful and only occasionally produced useful results. Performance metrics were calculated as described below:<\/p>\n
Our test set comprised 71 contract pages from the same government contract data source, but not yet seen by our model. All pages had both signature and non-signature handwriting present on the image.<\/p>\n
For each image, we defined two groups:<\/p>\n