{"id":2116,"date":"2017-01-11T04:00:00","date_gmt":"2017-01-11T12:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/index.php\/2017\/01\/11\/corpus-to-graph-building-a-pipeline-for-an-extracting-entity-relations-graph-from-a-corpus-of-documents\/"},"modified":"2020-03-19T10:21:26","modified_gmt":"2020-03-19T17:21:26","slug":"corpus-to-graph-building-a-pipeline-for-an-extracting-entity-relations-graph-from-a-corpus-of-documents","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/corpus-to-graph-building-a-pipeline-for-an-extracting-entity-relations-graph-from-a-corpus-of-documents\/","title":{"rendered":"Building a Pipeline for Extracting and Graphing Entities and Relations from a Corpus of Documents"},"content":{"rendered":"

Miroculus<\/a> is a startup developing a simple, quick and affordable blood test to diagnose cancer and other diseases at an early stage. Their test device can detect the existence of micro-RNAs (miRNAs)<\/a> in the patient\u2019s blood, which may correlate with a particular disease.<\/p>\n

Miroculus developed the loom.bio<\/a> tool that builds a visual graph according to the relations between Genes<\/strong>, miRNAs<\/strong> and Conditions<\/strong>, extracted from uncurated public articles (ie. Pubmed<\/a>, PMC<\/a>, etc.)<\/p>\n

Microsoft and Miroculus collaborated to build a pipeline to process a corpus of medical documents. The pipeline is a generalized solution that extracts entities from the documents, finds whether a relation exists between them, then stores the derived relations in a database representing a graph.<\/p>\n

The Problem<\/h2>\n

We wanted to build a corpus-to-graph pipeline that is:<\/p>\n

    \n
  1. Reusable across different corpora and domains<\/li>\n
  2. Scalable across different pipeline tasks<\/li>\n
  3. Easily integrated with GitHub for continuous deployment<\/li>\n
  4. Easily manageable<\/li>\n<\/ol>\n

    Design Architecture<\/h2>\n

    \"Diagram<\/p>\n

    Design Choices<\/h2>\n

    For this project, we extract relationships between miRNAs and genes, based on the connections that we find in medical research documents available on PubMed and PMC.<\/p>\n

    To extract entity relations from a single document, we perform the following sequence:<\/p>\n

      \n
    1. Split a document retrieved from the corpus into sentences<\/li>\n
    2. Extract miRNA and gene entities from each sentence<\/li>\n
    3. Use a binary classifier to discover relations between each pair of sentences and entities<\/li>\n
    4. Create an entry in the Graph DB for each related entity pair. The entry includes entities, classification, score, and reference sentence.<\/li>\n<\/ol>\n

      We\u2019ve developed a pipeline that uses a scheduled WebJob to periodically query for new documents in the corpus. This sequence is repeated for each document in the corpus.<\/p>\n

      Each job is implemented as an Azure App Service<\/a> feature called WebJobs<\/a>. Each WebJob<\/strong> listens to a queue, which represents the queue of job messages.<\/p>\n

      Azure Web Apps<\/strong> are easily integrated to GitHub<\/a>, and using queues is a good practice for supporting scalability.\nAdditionally, each WebJob<\/strong>, as part of Azure App Service<\/strong>, is scalable separately.<\/p>\n

      Domain-specific tasks were abstracted and represented as API calls, to keep the pipeline generic and reusable across different domains and corpora.<\/p>\n

      Next, we encountered two challenges:<\/p>\n

        \n
      1. How would we associate each Web App<\/strong> with the right job, given that we don\u2019t want to create separate repositories for each pipeline task?<\/li>\n
      2. How do we manage and monitor the pipeline?<\/li>\n<\/ol>\n

        We used Pipeline Management<\/a> to tackle the first challenge and Pipeline Deployment<\/a> to answer the second one.\nThe following sections detail how we addressed these challenges and integrated everything together into a holistic solution.<\/p>\n

        Pipeline Management<\/h2>\n

        The web-cli<\/a> is a great tool for controlling and monitoring your backend services and applications.\nIn addition to the built-in plugins for common tasks, the web-cli is easily extendable with plugins. One particularly useful plugin is for logging that enables you to query logs captured by a WebJob.<\/p>\n

        In our pipeline implementation, we extended the web-cli<\/strong> to support administration actions like rescore<\/strong> or update model<\/strong>.<\/p>\n

        This approach can be used to provide the user with a one-stop tool for managing the pipeline.<\/p>\n

        Domain Abstraction<\/h3>\n

        The pipeline supports two APIs that can be implemented according to a specific domain:<\/p>\n

          \n
        1. Entity extraction API: splits documents into sentences and extracts relevant entities.<\/li>\n
        2. Scoring API: detects relations between extracted entities.<\/li>\n<\/ol>\n

          We decided to separate these APIs from our pipeline for the following reasons:<\/p>\n

            \n
          1. Both APIs implement domain-specific logic.<\/li>\n
          2. Entity extraction<\/strong> and Scoring<\/strong> services use different technologies than those used in the pipeline.<\/li>\n
          3. Defining an API for these services enables our solution to be configurable.<\/li>\n
          4. Entity extraction<\/strong> and Scoring<\/strong> are processes that run on Linux, and we wanted to enable deployment of the pipeline to Azure App Services<\/strong>.<\/li>\n<\/ol>\n

            Pipeline Deployment<\/h3>\n

            The pipeline includes 4 Node.js WebJobs<\/strong>, with common dependencies and logic as well as two websites for web-cli<\/strong> and Graph API<\/strong>.<\/p>\n

            We wanted to leverage the Continuous Deployment<\/strong> feature using WebJobs<\/strong> but didn\u2019t want a separate repository for each pipeline role. Therefore, we figured out a way to deploy a single repository, with the implementation of all the pipeline\u2019s roles having each WebJob<\/strong> determine its role in the pipeline.<\/p>\n

            Here is an example of how web job runners are created according to a received parameter:\ncorpus-to-graph-pipeline\/lib\/runners\/continuous.js<\/a><\/p>\n

            \n
            var<\/span> roles<\/span> =<\/span> require<\/span>(<\/span>'..\/roles'<\/span>);<\/span>\r\n\/*...*\/<\/span>\r\nfunction<\/span> Runner<\/span>(<\/span>serviceName<\/span>,<\/span> config<\/span>,<\/span> options<\/span>)<\/span> {<\/span>\r\n  \r\n  \/\/ Initializing the relevant role accorindg to serviceName<\/span>\r\n  var<\/span> svc<\/span> =<\/span> new<\/span> roles<\/span>[<\/span>serviceName<\/span>](<\/span>config<\/span>,<\/span> options<\/span>);<\/span>\r\n\r\n  \/\/ This method is called periodically to check for messages in the queue<\/span>\r\n  function<\/span> checkInputQueue<\/span>()<\/span> {<\/span>\r\n\r\n    \/\/ Request a single message from the queue<\/span>\r\n    queueIn<\/span>.<\/span>getSingleMessage<\/span>(<\/span>function<\/span> (<\/span>err<\/span>,<\/span> message<\/span>)<\/span> {<\/span>\r\n\r\n      \/\/ Send message to be processed by the service<\/span>\r\n      return<\/span> svc<\/span>.<\/span>processMessage<\/span>(<\/span>msgObject<\/span>,<\/span> function<\/span> (<\/span>err<\/span>)<\/span> {<\/span>\r\n        \/*...*\/<\/span>\r\n      });<\/span>\r\n    });<\/span>\r\n  }<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n<\/div>\n
            \n
            var<\/span> continuousRunner<\/span> =<\/span> require<\/span>(<\/span>'corpus-to-graph-pipeline'<\/span>).<\/span>runners<\/span>.<\/span>continuous<\/span>;<\/span>\r\nvar<\/span> webJobName<\/span> =<\/span> process<\/span>.<\/span>env<\/span>.<\/span>PIPELINE_ROLE<\/span>;<\/span>\r\n\/*...*\/<\/span>\r\nfunction<\/span> startContinuousRunner<\/span>()<\/span> {<\/span>\r\n  \r\n  var<\/span> runnerInstance<\/span> =<\/span> new<\/span> continuousRunner<\/span>(<\/span>webJobName<\/span>,<\/span> \/*...*\/<\/span>);<\/span> \r\n  return<\/span> runnerInstance<\/span>.<\/span>start<\/span>(<\/span>function<\/span> (<\/span>err<\/span>)<\/span> {<\/span>\r\n    \/*...*\/<\/span>\r\n  });<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n<\/div>\n
            Our solution uses a generic node module and a sample solution that leverages it.<\/p>\n
            This is an example of how to start the runner with the relevant web job loaded from environment variables:\ncorpus-to-graph-genomics\/webjob\/continuous\/app.js<\/a><\/p>\n