{"id":4466,"date":"2017-07-05T11:00:01","date_gmt":"2017-07-05T18:00:01","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/?p=4466"},"modified":"2021-03-25T09:55:01","modified_gmt":"2021-03-25T16:55:01","slug":"imageregognitionandclassificationmlforautomating-receipt-processing","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/imageregognitionandclassificationmlforautomating-receipt-processing\/","title":{"rendered":"Automating Receipt Processing"},"content":{"rendered":"

Claiming expenses is usually a manual process. This project aims to improve the efficiency of receipt processing by looking into ways to automate this process.\u00a0<\/p>\n

This code story describes how we created a skeletal framework to achieve the following:<\/p>\n

    \n
  1. Classify the type of expense<\/li>\n
  2. Extract the amount spent<\/li>\n
  3. Extract the retailer (our example is limited to the most common retailers)<\/li>\n<\/ol>\n

    We found a few challenges in addressing these goals. For instance, the quality of an Optical Character Recognizer (OCR) is crucial to tasks like accurately extracting the information of interest and modeling text-based classifiers (e.g., the expense category classifier). In addition, we discovered some retailers use\u00a0logos instead of text for their names, which makes the extraction process more complex.<\/p>\n

    <\/p>\n

    The figure below shows the stages of addressing the goals of the framework, as well as the aforementioned challenges.<\/p>\n

    \"Image<\/p>\n

    First, a receipt is captured via a camera. Next, that image is passed to the Logo Recognizer<\/strong>, and the\u00a0Text Line Localizer<\/strong>, where the outputs (smaller chunks of texts)<\/span>\u00a0are then passed on to the\u00a0Optical Character Recognizer (OCR)<\/strong>\u00a0in the\u00a0Text Extractor<\/strong>.<\/p>\n

    The output of an OCR is a string of characters, which is then simultaneously passed to the\u00a0Text-based Retailer Recognizer<\/strong>,\u00a0Expense Category Recognizer<\/strong>, and\u00a0Total Amount Extractor<\/strong>. The\u00a0Retailer Recognizer<\/strong>\u00a0consists of two components: the\u00a0Logo Recognizer <\/strong>and the\u00a0Text-based Retailer Recognizer<\/strong>.<\/p>\n

    One of the predicted results from both the\u00a0Logo Recognizer <\/strong>and the\u00a0Text-based Retailer Recognizer <\/strong>is then selected.<\/p>\n

    The functions, methodologies, and code<\/a> used for each module are as follows:<\/p>\n

    Text Extractor<\/h2>\n

    The purpose of an OCR<\/strong> is to extract text out of an image. Here, we have experimented with\u00a0Microsoft Computer Vision OCR<\/a>, and open-source\u00a0Tesseract OCR<\/a>\u00a0(online\u00a0demo<\/a>). Both support multiple languages. There are other\u00a0OCRs<\/a>\u00a0out there which are mostly licensed. The table below shows\u00a0example output from both OCRs.<\/p>\n\n\n\n\n\n
    Receipt<\/th>\nMicrosoft-OCR<\/th>\nTesseract-OCR<\/th>\n<\/tr>\n<\/thead>\n
    \"Image<\/td>\n\"Image<\/td>\n\"Image<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Text Line Localizer<\/strong><\/h3>\n

    Text Connectionist Proposal Network (TCPN)<\/a>\u00a0and its\u00a0authors\u2019 Caffee implementation<\/a>, is used to break the whole image into smaller sub-images based on the existence of text.In other words, it locates lines of text in a natural image. It is a deep learning approach based on both recurrent neural networks and convolutional networks. Ideally, when an image is broken up into smaller regions before passing them into an OCR, it will help to boost the OCR’s performance. In a small set of samples, we found performance gains when feeding a whole image versus a sub-image into an OCR. The figure below shows the comparison with and without\u00a0Text Line Localizer<\/strong>, in combination with\u00a0Tesseract-OCR<\/a>.<\/p>\n\n\n\n\n\n
    Receipt<\/th>\nTesseract-OCR<\/th>\nTCPN + Tesseract-OCR Text Localization<\/th>\nTCPN + Tesseract-OCR Extracted Text<\/th>\n<\/tr>\n<\/thead>\n
    \"\"\"\"<\/td>\n\n
    tlttl vuxen 200.000r<\/span><\/span>\r\nARLANDA m 9a-rtomrttrSlOg-<\/span><\/span>\r\n_ 20N1<\/span><\/span>\r\n'CT, I Zen 1 \" Arhnda 0<\/span><\/span>\r\nKontant<\/span><\/span><\/pre>\n<\/td>\n
    		\"\"\"\"<\/td>\n\"\"\"\"<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    The authors claimed that the algorithm works reliably on multi-scale and multi- language texts without further post-processing and computationally efficient. We have made available\u00a0the code for deploying this Caffe model in the Windows environment<\/a>, specifically in an\u00a0Azure Data Science Virtual Machine<\/a>.<\/p>\n

    If you are interested in using both\u00a0Text Line Localizer<\/strong>\u00a0and\u00a0OCR<\/strong>\u00a0(in this example,\u00a0Tesseract-OCR<\/a>) in a sequential manner, wrapped in a web API, please refer to\u00a0Node.js web server with ML model<\/em>\u00a0(TODO @Eero)<\/a>.<\/p>\n

    Information of Interest Extractor<\/h2>\n

    This consists of a Retailer Recognizer, Expense Type Recognizer and Total Amount Extractor.<\/p>\n

    Retailer Recognizer (Logo Recognizer)<\/h3>\n

    In this example, we use Microsoft Cognitive Services’\u00a0Custom Vision<\/a>\u00a0Service to build a custom model for recognising retailers based on the look of a receipt and\/or a logo. Custom Vision allows you to easily customize your own computer vision models to fit your unique use case. It requires a few dozen samples of labeled images for each class. We trained a model using the entire receipt images in our example. It is possible to only provide a specific region during training; for instance, if most of the time the logo will be at the top of the receipt, then we can use only that part for model-building. When it comes to prediction, either just the top region\u00a0or the whole receipt can be fed into the predictor. The figure below shows the overall performance and some experimentation results with various classes and the respective number of samples uploaded to\u00a0Custom Vision \u2014 namely, rail (33),\u00a0bandq\u00a0(14),\u00a0pizzaexpress\u00a0(18),\u00a0walmart\u00a0(34) and\u00a0asda\u00a0(26).<\/p>\n

    \"Image<\/p>\n\n\n\n\n\n\n\n\n\n\n\n
    <\/th>\nExample results<\/th>\n<\/th>\n<\/th>\n<\/tr>\n<\/thead>\n
    1.<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n<\/tr>\n
    2.<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n<\/tr>\n
    3.<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n<\/tr>\n
    4.<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n<\/tr>\n
    5.<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n<\/tr>\n
    6.<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n<\/tr>\n
    7.<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    The table above shows some example\u00a0exemplar results. This model classifies well most of the time for receipts from known retailers. It is able to distinguish a receipt from a non-receipt (see row 6) such as a zebra. Note the confident probability scores shown. In row 2 and 4, there is a test image with multiple\u00a0bandq<\/em>\u00a0receipts and a test image with multiple\u00a0walmart<\/em>\u00a0test receipts. In this example, multiple Walmart receipts appear in the same image and are correctly classified as\u00a0walmart<\/em>, whilst a group of B&Q receipts in the same image are not correctly classified. This sample is just to show how the model will behave in corner cases like this. In practice, restrictions can be put in place to help, such as limiting it to one single receipt at a time. Row 7 shows receipts from retailers of which the model has no knowledge. Unfortunately, the classifier is confused when a receipt which does not belong to any of the known classes is tested.<\/p>\n

    To address the issue above, try adding a class called\u00a0others<\/em>, which will be a collection of receipts that have no logos, as well as any receipts that are not from the intended 5 classes. How to decide between these two options will depend on application-specific requirements.<\/p>\n

    The figures below shows the performance of a different model which has the class\u00a0others <\/em>incorporated.<\/p>\n

    \"Image<\/p>\n

    In this example, 76 samples were uploaded to\u00a0Custom Vision<\/a>\u00a0for the model building.<\/p>\n\n\n\n\n\n
    Example results<\/th>\n<\/th>\n<\/th>\n<\/tr>\n<\/thead>\n
    \"jpg:<\/td>\n\"jpg:<\/td>\n\"jpg:<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Note that the model\u2019s confidence in ruling out those receipts without logos<\/span> as one of the 5 classes of receipts with logos. The first and third receipts are confidently classified as\u00a0others<\/em>. However, consider the test image in the middle: while it is predicted as\u00a0others<\/em>\u00a0with the highest probability, the score for\u00a0pizzaexpress<\/em>\u00a0is rather high too. This confusion could perhaps be mitigated with increased variety of training samples for the class\u00a0others<\/em>.<\/p>\n

    Text-based Retailer Recognizer<\/h3>\n

    In contrast to the\u00a0Logo Recognizer<\/strong>, this module recognises the retailer by the text extracted from the receipt. As per the\u00a0Logo Recognizer<\/strong>, instead of using the whole receipt, it is possible just to use certain portion of text extracted to train a text-based retailer recognizer. This is essentially a text classification problem, similar to the\u00a0Expense Recognizer<\/strong>.<\/p>\n

    Selector<\/h3>\n

    In this example, we naively select a result from either the\u00a0Logo Recognizer<\/strong>\u00a0or the\u00a0Text-Based Retailer Recognizer<\/strong>\u00a0(whichever has the highest probability). There are better ways to select a more confident result, such as weighted or non-weighted, fuzzy method, linear or non-linear combination, intuition-based and so on. While this topic is beyond the scope of this post, it will become increasingly important when more classifiers are applied in an ensemble manner. This lightweight, high-level\u00a0article<\/a>\u00a0touches on some basic methods.<\/p>\n

    Expense Recognizer<\/h3>\n

    Its purpose is to recognize the type of expense, like accommodation, meal, transport etc. This is based on text occurring within the receipt. It is trained within\u00a0Azure ML Studio<\/a>\u00a0using the typical process of text classification. In this example, the\u00a0Expense Recognizer<\/strong>\u00a0is based on\u00a0Using Microsoft Cognitive Services within Azure ML Studio to Predict Expense Type from Receipts<\/em><\/a>. While this is a simplistic approach, many variations of text processing can be applied. Again, the discussion of what are the best approaches or algorithms is beyond the scope of this post. This approach is similar to that of the\u00a0Text-based Retailer Recognizer<\/strong>.<\/p>\n

    Total Amount Extractor<\/h3>\n

    At the time of writing, this extractor is naively extracting numbers that show the format of monetary value, and assume the largest value is the total amount spent. Most of the time it works well with credit card purchases but will encounter issues if it is a cash purchase.<\/p>\n

    These modules are tied together via Azure Functions, Azure Storage and Azure Queue. The figure below shows the architectural diagram. The article\u00a0Using Azure Functions to enable processing of Receipt Images with OCR<\/em><\/a>\u00a0highlights the end-to-end\u00a0implementation\u00a0and the benefits of using such architecture.<\/p>\n

    \"png:<\/p>\n

    All the modules within\u00a0Information of Interest Extractor\u00a0are implemented within\u00a0Azure ML Studio<\/a>\u00a0for easy deployment as a web service, except for the\u00a0Logo Recognizer, whose the model is built using\u00a0Custom Vision\u00a0resulting in a web service. This API is then called from within\u00a0Azure ML Studio. Implementation of all the modules stated in the Information of Interest Extractor\u00a0can be used within the Execute Python Script\u00a0module as part of an Azure ML Studio experiment.<\/p>\n

    Further Work<\/h2>\n

    Here are some other things we have tried but need further work.<\/p>\n

    Fast-R-CNN (CNTK implementation)<\/h3>\n

    We found that if we can increase the reliability of OCR, the better quality of text extracted may help to improve the information extraction process.\u00a0CNTK\u2019s Fast-R-CNN<\/a>\u00a0was initially tested in an attempt to create both word- and character-level recognition. Note that there is a\u00a0Faster-R-CNN<\/strong>\u00a0<\/a>with similar function.<\/p>\n

    A note on Selective Search<\/h4>\n

    At the time of testing (May 2017), we found that first part of the algorithm, Selective Search (SS),\u00a0is suitable for images of natural scenes with rich colour and of a certain minimum size. However, it struggles to propose new Regions of Interest (ROI) when images mainly consist of black characters on a white-ish background (that is, a lack of colour richness and complex fine patterns in a small region). Meanwhile, there are possible alternatives to Selective Search. The simplest way is naively shifting the bounding box (labelled\u00a0box)\u202fto the top, bottom, left, or right by a small amount. This approach generates multiple regions of interests, before feeding the ROIs into the network.<\/p>\n

    Conclusion<\/h2>\n

    Here, we show an example technique where receipt processing can be automated using a combination of text extraction and image recognition techniques, together with some challenges. If you have any comments, please share with us in the comments below.<\/p>\n

    References<\/h2>\n