{"id":14115,"date":"2022-05-13T13:40:24","date_gmt":"2022-05-13T20:40:24","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cse\/?p=14115"},"modified":"2023-06-19T11:14:00","modified_gmt":"2023-06-19T18:14:00","slug":"a-search-solution-for-video-archive","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/a-search-solution-for-video-archive\/","title":{"rendered":"A Search Solution For Video Archive"},"content":{"rendered":"<h3 aria-level=\"2\"><span data-contrast=\"none\">Background<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h3>\n<p><span class=\"TextRun SCXW228273280 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW228273280 BCX0\">Azure Video Analyzer<\/span><span class=\"NormalTextRun SCXW228273280 BCX0\"> enables <\/span><span class=\"NormalTextRun SCXW228273280 BCX0\">developers <\/span><span class=\"NormalTextRun SCXW228273280 BCX0\">to <\/span><span class=\"NormalTextRun SCXW228273280 BCX0\">quickly build an AI-powered video analytics solution to extract actionable insights from videos, whether stored or <\/span><span class=\"NormalTextRun ContextualSpellingAndGrammarErrorV2Themed SCXW228273280 BCX0\">streaming<\/span><span class=\"NormalTextRun SCXW228273280 BCX0\">. <\/span><span class=\"NormalTextRun SCXW228273280 BCX0\">It can be<\/span><span class=\"NormalTextRun SCXW228273280 BCX0\"> used to<\/span> <span class=\"NormalTextRun SCXW228273280 BCX0\">enhance workplace safety, in-store experiences, digital asset management, media archives, and more by <\/span><span class=\"NormalTextRun SCXW228273280 BCX0\">assisting<\/span><span class=\"NormalTextRun SCXW228273280 BCX0\"> in understanding <\/span><span class=\"NormalTextRun SCXW228273280 BCX0\">audio and visual video content without video <\/span><span class=\"NormalTextRun SCXW228273280 BCX0\">or <\/span><span class=\"NormalTextRun SCXW228273280 BCX0\">machine learning <\/span><span class=\"NormalTextRun SCXW228273280 BCX0\">expertise<\/span><span class=\"NormalTextRun SCXW228273280 BCX0\"> [1].<\/span><\/span><span class=\"EOP SCXW228273280 BCX0\" data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0 In this solution we use <span class=\"TextRun SCXW228273280 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW228273280 BCX0\">Azure Video Analyzer and Azure Search to be able to retrieve desired moments in our video archive.\u00a0\u00a0<\/span><\/span><\/span><\/p>\n<p>&nbsp;<\/p>\n<h3 aria-level=\"2\"><span data-contrast=\"none\">Challenges and Objectives<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:276}\">\u00a0<\/span><\/h3>\n<p><span class=\"TextRun SCXW121852879 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW121852879 BCX0\">Our <\/span><span class=\"NormalTextRun ContextualSpellingAndGrammarErrorV2Themed SCXW121852879 BCX0\">customers<\/span> <span class=\"NormalTextRun SCXW121852879 BCX0\">needed <\/span><span class=\"NormalTextRun SCXW121852879 BCX0\">to be able to search their <\/span><span class=\"NormalTextRun SCXW121852879 BCX0\">video archives<\/span><span class=\"NormalTextRun SCXW121852879 BCX0\"> to find <\/span><span class=\"NormalTextRun SCXW121852879 BCX0\">result<\/span><span class=\"NormalTextRun SCXW121852879 BCX0\">s<\/span><span class=\"NormalTextRun SCXW121852879 BCX0\"> related to a video <\/span><span class=\"NormalTextRun SCXW121852879 BCX0\">for<\/span><span class=\"NormalTextRun SCXW121852879 BCX0\"> a <\/span><span class=\"NormalTextRun SCXW121852879 BCX0\">given <\/span><\/span><span class=\"TrackedChange SCXW121852879 BCX0\"><span class=\"TextRun SCXW121852879 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW121852879 BCX0\">moment<\/span><\/span><\/span><span class=\"TextRun SCXW121852879 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW121852879 BCX0\">. This post provides a solution to be able to search for specific features and retrieve the moments related to the searched query.<\/span><\/span><\/p>\n<p><span class=\"TextRun SCXW21136728 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW21136728 BCX0\">For the purposes of this post, <\/span><span class=\"NormalTextRun SCXW21136728 BCX0\">a <\/span><\/span><span class=\"TextRun SCXW21136728 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW21136728 BCX0\">moment<\/span><\/span><span class=\"TextRun SCXW21136728 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW21136728 BCX0\"> covers n number of seconds.<\/span><span class=\"NormalTextRun SCXW21136728 BCX0\"> An example of a <\/span><\/span><span class=\"TextRun SCXW21136728 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW21136728 BCX0\">moment<\/span><\/span><span class=\"TextRun SCXW21136728 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"> <span class=\"NormalTextRun SCXW21136728 BCX0\">query <\/span><span class=\"NormalTextRun SCXW21136728 BCX0\">is <em>\u201cfind moments where <strong>Jean Claude<\/strong>\u00a0appeared in <strong>Friends <\/strong><\/em><\/span><em><span class=\"NormalTextRun SCXW21136728 BCX0\">tv <\/span><span class=\"NormalTextRun SCXW21136728 BCX0\">show<\/span><span class=\"NormalTextRun SCXW21136728 BCX0\">.\u201d<\/span><\/em><\/span><\/p>\n<p><span class=\"TextRun SCXW140809292 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW140809292 BCX0\">Additionally, our customer <\/span><span class=\"NormalTextRun SCXW140809292 BCX0\">required<\/span><span class=\"NormalTextRun SCXW140809292 BCX0\"> support for custom features in Azure Video Analyzer.<\/span><\/span><span class=\"EOP SCXW140809292 BCX0\" data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559685&quot;:0,&quot;335559737&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<h4>The Challenges:<\/h4>\n<ul style=\"font-weight: 400;\">\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"3\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559682&quot;:3,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"none\">Azure Video Analyzer receive a video file and provides <\/span><span data-contrast=\"none\">one JSON file containing extracted features for that video (e.g., one file containing all the features for a 00:20:10 show). Hence if we use that file as a record in our search index, we cannot resolve the mentioned query without post-processing steps to find the desired moments. <\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559737&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"3\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559682&quot;:3,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"2\" data-aria-level=\"1\"><span data-contrast=\"none\">Post-processing of large records in order to find the relevant moments is too costly and inefficient.<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"3\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559682&quot;:3,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"2\" data-aria-level=\"1\"><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559737&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">Azure Video Analyzer, natively does not support custom features.<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"none\">In this post, we will discuss our experience using Azure Video Analyzer to accomplish this objective in three steps:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559685&quot;:0,&quot;335559737&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<ol>\n<li><span data-contrast=\"none\">Create a search index, with flexibility to accommodate custom features.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559737&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li><span data-contrast=\"none\">Upload <\/span><i><span data-contrast=\"none\">moments<\/span><\/i><span data-contrast=\"none\"> to the index.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559737&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li><span data-contrast=\"none\">Establish search queries to retrieve related <\/span><i><span data-contrast=\"none\">moments<\/span><\/i> <span data-contrast=\"none\">from the index.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559737&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<\/ol>\n<h3 aria-level=\"2\"><span data-contrast=\"none\">Solution<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h3>\n<p><span data-contrast=\"auto\">We resolved the challenges mentioned in the previous section in two ways:<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<ol>\n<li data-leveltext=\"%1-\" data-font=\"Calibri\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1-&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Create a search index with all existing Azure Video Analyzer features and flexibility to have more fields as custom features.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<\/ol>\n<ol>\n<li data-leveltext=\"%1-\" data-font=\"Calibri\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:0,&quot;335559684&quot;:-1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1-&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"2\" data-aria-level=\"1\"><span data-contrast=\"auto\">Parse the Azure Video Analyzer JSON output file into small intervals (n seconds) and feed them into the Azure search index.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<\/ol>\n<p><span class=\"TextRun SCXW101029199 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW101029199 BCX0\">Ea<\/span><span class=\"NormalTextRun SCXW101029199 BCX0\">ch record in the Azure Search index is equivalent to <\/span><\/span><span class=\"TrackedChange SCXW101029199 BCX0\"><span class=\"TextRun SCXW101029199 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW101029199 BCX0\">n<\/span><\/span><\/span><span class=\"TextRun SCXW101029199 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW101029199 BCX0\"> seconds of a video. When the end-user searches for a query, the most relevant <\/span><span class=\"NormalTextRun SCXW101029199 BCX0\">record (for instance moments 00:11:30 to 00:11:40 of a video) will be retrieved.\u00a0<\/span> <span class=\"NormalTextRun SCXW101029199 BCX0\">In the next section we <\/span><span class=\"NormalTextRun SCXW101029199 BCX0\">provide<\/span><span class=\"NormalTextRun SCXW101029199 BCX0\"> a deeper <\/span><span class=\"NormalTextRun SCXW101029199 BCX0\">explanation of<\/span><span class=\"NormalTextRun SCXW101029199 BCX0\"> how we <\/span><span class=\"NormalTextRun SCXW101029199 BCX0\">accomplished<\/span><span class=\"NormalTextRun SCXW101029199 BCX0\"> our goals.<\/span><\/span><span class=\"EOP SCXW101029199 BCX0\" data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<h3 aria-level=\"2\"><span data-contrast=\"none\">Architecture deep-dive<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h3>\n<p><span data-contrast=\"auto\">We assume that you have already processed the video files using <\/span><a href=\"https:\/\/azure.microsoft.com\/en-us\/products\/video-analyzer\/\"><span data-contrast=\"none\">Azure Video Analyzer<\/span><\/a><span data-contrast=\"auto\"> and uploaded the output files into a container in a storage account.\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Here is the system sequence diagram of the Azure Video Indexer Searchable moments.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2022\/01\/Parser.png\"><img decoding=\"async\" class=\"wp-image-14116 aligncenter\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2022\/01\/Parser.png\" alt=\"Image Parser\" width=\"651\" height=\"485\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2022\/01\/Parser.png 882w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2022\/01\/Parser-300x223.png 300w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2022\/01\/Parser-768x572.png 768w\" sizes=\"(max-width: 651px) 100vw, 651px\" \/><\/a><\/p>\n<h3 aria-level=\"3\"><span data-contrast=\"none\">Create Search index:<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h3>\n<p><span class=\"TextRun SCXW133488416 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW133488416 BCX0\">We should have a search index that <\/span><span class=\"NormalTextRun SCXW133488416 BCX0\">supports<\/span><span class=\"NormalTextRun SCXW133488416 BCX0\"> all video indexer features. <\/span><span class=\"NormalTextRun SCXW133488416 BCX0\">Depending on<\/span><span class=\"NormalTextRun SCXW133488416 BCX0\"> our <\/span><span class=\"NormalTextRun SCXW133488416 BCX0\">use case<\/span><span class=\"NormalTextRun SCXW133488416 BCX0\">, we may need to add <\/span><span class=\"NormalTextRun SCXW133488416 BCX0\">additional<\/span><span class=\"NormalTextRun SCXW133488416 BCX0\"> fields <\/span><span class=\"NormalTextRun SCXW133488416 BCX0\">to<\/span><span class=\"NormalTextRun SCXW133488416 BCX0\"> our search index to accommodate those custom features not<\/span><span class=\"NormalTextRun SCXW133488416 BCX0\"> provided by the Azure Video Indexer.<\/span><\/span><span class=\"EOP SCXW133488416 BCX0\" data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<h3 aria-level=\"3\"><span data-contrast=\"none\">Parse Video Analyzer Output to Moments:<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h3>\n<p><span data-contrast=\"auto\">Video Indexer process shows and generates one JSON file per video.\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">First, we need to store those JSON files into a container in our azure storage account.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Next, we need to parse these JSON files into <\/span><i><span data-contrast=\"auto\">n<\/span><\/i><span data-contrast=\"auto\"> intervals (e.g., n=10 seconds interval). <\/span><span data-contrast=\"auto\">This means that if a video duration is <\/span><b><span data-contrast=\"auto\">00:20:10,<\/span><\/b><span data-contrast=\"auto\"> using <\/span><b><span data-contrast=\"auto\">10<\/span><\/b><span data-contrast=\"auto\"> seconds intervals, we will have <\/span><b><span data-contrast=\"auto\">121<\/span><\/b><span data-contrast=\"auto\"> documents,\u00a0each covering 10 seconds of the show.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2022\/01\/Screen-Shot-2022-01-24-at-3.51.47-PM.png\"><img decoding=\"async\" class=\"wp-image-14117 size-full aligncenter\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2022\/01\/Screen-Shot-2022-01-24-at-3.51.47-PM.png\" alt=\"Image Screen Shot 2022 01 24 at 3 51 47 PM\" width=\"749\" height=\"324\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2022\/01\/Screen-Shot-2022-01-24-at-3.51.47-PM.png 749w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2022\/01\/Screen-Shot-2022-01-24-at-3.51.47-PM-300x130.png 300w\" sizes=\"(max-width: 749px) 100vw, 749px\" \/><\/a><\/p>\n<p><span class=\"TextRun SCXW163738950 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW163738950 BCX0\">Finally, we will upload <\/span><span class=\"NormalTextRun SCXW163738950 BCX0\">the <\/span><span class=\"NormalTextRun SCXW163738950 BCX0\">intervals<\/span><span class=\"NormalTextRun SCXW163738950 BCX0\"> defined above<\/span><span class=\"NormalTextRun SCXW163738950 BCX0\"> (e.g., 121 records of a show having duration of 00:20:10 with n=10 seconds interval) into the created search index. You can find the architecture of the Azure Video Indexer<\/span><span class=\"NormalTextRun SCXW163738950 BCX0\"> Searchable<\/span><span class=\"NormalTextRun SCXW163738950 BCX0\"> moments <\/span><span class=\"NormalTextRun SCXW163738950 BCX0\">library<\/span><span class=\"NormalTextRun SCXW163738950 BCX0\"> below<\/span><span class=\"NormalTextRun SCXW163738950 BCX0\">.<\/span><\/span><span class=\"EOP SCXW163738950 BCX0\" data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2022\/01\/Azure-Vdieo-Analyzer-Searchable-Moments.png\"><img decoding=\"async\" class=\"aligncenter wp-image-14118 size-large\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2022\/01\/Azure-Vdieo-Analyzer-Searchable-Moments-1024x658.png\" alt=\"Image Azure Video Analyzer Searchable Moments\" width=\"640\" height=\"411\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2022\/01\/Azure-Vdieo-Analyzer-Searchable-Moments-1024x658.png 1024w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2022\/01\/Azure-Vdieo-Analyzer-Searchable-Moments-300x193.png 300w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2022\/01\/Azure-Vdieo-Analyzer-Searchable-Moments-768x494.png 768w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2022\/01\/Azure-Vdieo-Analyzer-Searchable-Moments.png 1440w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><span class=\"TextRun SCXW215806600 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW215806600 BCX0\" data-ccp-charstyle=\"normaltextrun\" data-ccp-charstyle-defn=\"{&quot;ObjectId&quot;:&quot;fae65bdb-e361-4885-9358-a0f7d1e29220|44&quot;,&quot;ClassId&quot;:1073872969,&quot;Properties&quot;:[469775450,&quot;normaltextrun&quot;,201340122,&quot;1&quot;,134233614,&quot;true&quot;,469778129,&quot;normaltextrun&quot;,335572020,&quot;1&quot;,469778324,&quot;Default Paragraph Font&quot;]}\">Now that we have uploaded our records into the <\/span><span class=\"NormalTextRun SCXW215806600 BCX0\" data-ccp-charstyle=\"normaltextrun\">A<\/span><span class=\"NormalTextRun SCXW215806600 BCX0\" data-ccp-charstyle=\"normaltextrun\">zure<\/span><span class=\"NormalTextRun SCXW215806600 BCX0\" data-ccp-charstyle=\"normaltextrun\"> search index, we can find the right moments in the show related to our search query.<\/span><span class=\"NormalTextRun SCXW215806600 BCX0\" data-ccp-charstyle=\"normaltextrun\"> Here are some of the examples we could address using <\/span><span class=\"NormalTextRun SCXW215806600 BCX0\" data-ccp-charstyle=\"normaltextrun\">this<\/span> <span class=\"NormalTextRun SCXW215806600 BCX0\" data-ccp-charstyle=\"normaltextrun\">method<\/span><span class=\"NormalTextRun SCXW215806600 BCX0\" data-ccp-charstyle=\"normaltextrun\">:<\/span><\/span><span class=\"EOP SCXW215806600 BCX0\" data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559685&quot;:0,&quot;335559737&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p><span class=\"TextRun SCXW17880821 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW17880821 BCX0\"><strong>Example 1:<\/strong> <em>&#8220;Find the moments where <strong>&#8216;<\/strong><\/em><\/span><\/span><em><strong><span class=\"TextRun MacChromeBold SCXW17880821 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW17880821 BCX0\">Jennifer Aniston<\/span><\/span><\/strong><span class=\"TextRun SCXW17880821 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW17880821 BCX0\"><strong>&#8216;<\/strong> in Friends <\/span><span class=\"NormalTextRun SCXW17880821 BCX0\">TV<\/span><span class=\"NormalTextRun SCXW17880821 BCX0\"> show is <strong>happy<\/strong>&#8220;<\/span><\/span><\/em><span class=\"EOP SCXW17880821 BCX0\" data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p><strong>Search Query:<\/strong><\/p>\n<pre class=\"prettyprint\">{ \r\n  \"search\": \"Jennifer Aniston\", \r\n  \"searchFields\": \"faces\/face, name\", \r\n  \"filter\": \"search.ismatchscoring('friends')and (emotions\/any(em: \r\n    em\/emotion eq 'Joy')) \" \r\n}<\/pre>\n<p><span class=\"TextRun SCXW17880821 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW17880821 BCX0\"><strong>Example 2:<\/strong> <em><span class=\"TextRun SCXW193680479 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW193680479 BCX0\">&#8220;Find the moments in any show where the transcript is <strong>&#8216;<\/strong><\/span><\/span><strong><span class=\"TextRun MacChromeBold SCXW193680479 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW193680479 BCX0\">getting old<\/span><\/span><\/strong><span class=\"TextRun SCXW193680479 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW193680479 BCX0\"><strong>&#8216;<\/strong>&#8220;<\/span><\/span><\/em><\/span><\/span><\/p>\n<p><strong>Search Query:\u00a0<\/strong><\/p>\n<pre class=\"prettyprint\">{\r\n  \"search\": \"getting old\",\r\n  \"searchFields\": \"transcripts\/transcript\"\r\n}<\/pre>\n<p><span class=\"TextRun SCXW17880821 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW17880821 BCX0\"><strong>Example 3:<\/strong> <em><span class=\"TextRun SCXW193680479 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW193680479 BCX0\">&#8220;<span class=\"TextRun SCXW62070756 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW62070756 BCX0\">Find the moments in <strong>Friends show<\/strong> where &#8216;<\/span><\/span><strong><span class=\"TextRun MacChromeBold SCXW62070756 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW62070756 BCX0\">Jean Claude<\/span><\/span><\/strong><span class=\"TextRun SCXW62070756 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW62070756 BCX0\"><strong>&#8216;<\/strong> Appears&#8221;<\/span><\/span><\/span><\/span><\/em><\/span><\/span><\/p>\n<p><strong>Search Query:\u00a0<\/strong><\/p>\n<pre class=\"prettyprint\">{  \r\n  \"search\": \"Jean Claude\",\r\n  \"searchFields\": \"faces\/face, namedPeople\/namedPerson\",\r\n  \"filter\": \"search.ismatchscoring('friends') \",\r\n}<\/pre>\n<h3><span class=\"TextRun SCXW95095058 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW95095058 BCX0\" data-ccp-parastyle=\"heading 2\">Summary<\/span><\/span><\/h3>\n<p><span data-contrast=\"auto\">Azure Video Indexer provides one file containing features extracted from that video. However, if we upload that file into our search index, we cannot retrieve the moments which correspond to our search query. Instead, we will get the video itself.\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">In this blog post we propose a method that helps us search for moments in a video by splitting a video into small intervals of <\/span><i><span data-contrast=\"auto\">n<\/span><\/i><span data-contrast=\"auto\"> seconds (e.g., n=10 seconds). Then we will upload those records into an azure search index which is compatible with our video indexer intervals. You can find the sample code in <a href=\"https:\/\/github.com\/Azure-Samples\/SearchForMomentsInVideos\">this repository<\/a>.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">As a result, when we search for our desired query, the search engine will retrieve the most relevant record (containing n seconds of a video, e.g., n= 10 seconds).<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<h3><span class=\"TextRun SCXW266123813 BCX0\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW266123813 BCX0\" data-ccp-parastyle=\"heading 2\">The Team<\/span><\/span><\/h3>\n<p><span data-contrast=\"auto\">Below is the Microsoft team that worked with me on this project.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" aria-setsize=\"-1\" data-aria-posinset=\"4\" data-aria-level=\"1\"><a href=\"https:\/\/www.linkedin.com\/in\/mokarian\/\"><span data-contrast=\"none\">Maysam Mokarian<\/span><\/a><span data-contrast=\"none\">\u00a0,\u00a0<\/span><a href=\"mailto:mamokari@microsoft.com\"><span data-contrast=\"none\">mamokari@microsoft.com<\/span><\/a><span data-contrast=\"none\">\u00a0\u00a0\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" aria-setsize=\"-1\" data-aria-posinset=\"5\" data-aria-level=\"1\"><a href=\"https:\/\/www.linkedin.com\/in\/jitghosh\/\"><span data-contrast=\"none\">Jit Ghosh<\/span><\/a><span data-contrast=\"none\">\u00a0,\u00a0<\/span><a href=\"mailto:pghosh@microsoft.com\"><span data-contrast=\"none\">pghosh@microsoft.com<\/span><\/a><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" aria-setsize=\"-1\" data-aria-posinset=\"6\" data-aria-level=\"1\"><a href=\"https:\/\/www.linkedin.com\/in\/msolhab\/\"><span data-contrast=\"none\">Mona Soliman Habib<\/span><\/a><span data-contrast=\"none\">\u00a0,\u00a0<\/span><a href=\"mailto:Mona.Habib@microsoft.com\"><span data-contrast=\"none\">Mona.Habib@microsoft.com<\/span><\/a><span data-contrast=\"none\">\u00a0\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" aria-setsize=\"-1\" data-aria-posinset=\"7\" data-aria-level=\"1\"><a href=\"https:\/\/www.linkedin.com\/in\/jacobspoelstra\/\"><span data-contrast=\"none\">Jacob Spoelstra<\/span><\/a><span data-contrast=\"none\"> , <\/span><a href=\"mailto:Jacob.Spoelstra@microsoft.com\"><span data-contrast=\"none\">Jacob.Spoelstra@microsoft.com<\/span><\/a> <span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" aria-setsize=\"-1\" data-aria-posinset=\"8\" data-aria-level=\"1\"><a href=\"https:\/\/www.linkedin.com\/in\/activenick\/\"><span data-contrast=\"none\">Nick Landry<\/span><\/a><span data-contrast=\"none\"> , <\/span><a href=\"mailto:Nick.Landry@microsoft.com\"><span data-contrast=\"none\">Nick.Landry@microsoft.com<\/span><\/a> <span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" aria-setsize=\"-1\" data-aria-posinset=\"9\" data-aria-level=\"1\"><a href=\"https:\/\/www.linkedin.com\/in\/annazietlow\/\"><span data-contrast=\"none\">Anna Zietlow<\/span><\/a><span data-contrast=\"none\"> , <\/span><a href=\"mailto:Anna.Zietlow@microsoft.com\"><span data-contrast=\"none\">Anna.Zietlow@microsoft.com<\/span><\/a><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" aria-setsize=\"-1\" data-aria-posinset=\"10\" data-aria-level=\"1\"><a href=\"https:\/\/www.linkedin.com\/in\/samantha-rouphael\/\"><span data-contrast=\"none\">Samantha Rouphael<\/span><\/a><span data-contrast=\"none\"> , <\/span><a href=\"mailto:saroup@microsoft.com\"><span data-contrast=\"none\">saroup@microsoft.com<\/span><\/a><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" aria-setsize=\"-1\" data-aria-posinset=\"10\" data-aria-level=\"1\"><a href=\"https:\/\/www.linkedin.com\/in\/chrisdmayo\/\">Chris Mayo<\/a> , <a href=\"mailto:cmayo@microsoft.com\">cmayo@microsoft.com<\/a><\/li>\n<\/ul>\n<h3 aria-level=\"2\"><span data-contrast=\"none\">References<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h3>\n<ol>\n<li><span data-contrast=\"auto\"><a href=\"https:\/\/azure.microsoft.com\/en-us\/products\/video-analyzer\">Azure Video Analyzer<\/a><\/span>\u00a0<span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li><a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/search\/\">Azure Search<\/a><\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>Background\u00a0\u00a0 Azure Video Analyzer enables developers to quickly build an AI-powered video analytics solution to extract actionable insights from videos, whether stored or streaming. It can be used to enhance workplace safety, in-store experiences, digital asset management, media archives, and more by assisting in understanding audio and visual video content without video or machine learning [&hellip;]<\/p>\n","protected":false},"author":53988,"featured_media":14365,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-14115","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cse"],"acf":[],"blog_post_summary":"<p>Background\u00a0\u00a0 Azure Video Analyzer enables developers to quickly build an AI-powered video analytics solution to extract actionable insights from videos, whether stored or streaming. It can be used to enhance workplace safety, in-store experiences, digital asset management, media archives, and more by assisting in understanding audio and visual video content without video or machine learning [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/14115","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/53988"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=14115"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/14115\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/14365"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=14115"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=14115"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=14115"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}