{"id":13826,"date":"2021-09-20T08:59:12","date_gmt":"2021-09-20T15:59:12","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cse\/?p=13826"},"modified":"2023-06-19T10:50:47","modified_gmt":"2023-06-19T17:50:47","slug":"building-an-action-detection-scoring-pipeline-for-digital-dailies","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/building-an-action-detection-scoring-pipeline-for-digital-dailies\/","title":{"rendered":"Building an Action Detection Scoring Pipeline for Digital Dailies"},"content":{"rendered":"

Introduction<\/span><\/h2>\n

In media companies, like WarnerMedia, footage filmed for the entire day include \u2018takes\u2019 of various scenes or footage types as well as footage before and after each take. This footage filmed each day is known as \u2018digital dailies\u2019.<\/p>\n

Archival of digital dailies is a manual and time-consuming process. When digital dailies are produced all the data is permanently archived, however there is a portion of that content that should either be long-term archived or completely discarded. The problem here is two-pronged: 1. It\u2019s an expensive manual process to identify which portions of content can be archived and\/or discarded and 2. There is a cost associated with storing unnecessary content, especially when we are dealing with terabytes and petabytes of data.<\/p>\n

This project is a co-engineering collaboration between WarnerMedia and Microsoft\u2019s Commercial Software Engineering for identifying action and cut sequences within media for archival purposes.<\/p>\n

Goals<\/span><\/h2>\n

Our primary goal was to use Machine Learning to identify archival content from WarnerMedia digital dailies, that could be either long-term archived or discarded, to reduce storage costs.<\/p>\n

Exploring the Possibilities<\/span><\/h2>\n

We focused on identifying \u201caction\u201d and \u201ccut\u201d using visual and audio cues in the footage as a means of demarcating the content to identify portions for long-term storage and portions that can be discarded.<\/p>\n

When an \u201caction\u201d event has been identified, and up to a \u201ccut\u201d event, that\u2019s content that should be kept. When \u201ccut\u201d happens, and up to the next \u201caction\u201d that\u2019s content that could be archived or discarded.<\/p>\n

<\/p>\n

To accurately detect \u201caction\u201d and \u201ccut\u201d, the Data Science team iterated through many different services and experimentations.<\/p>\n

Visual Exploration:<\/h4>\n

<\/p>\n

The goal with the visual exploration is to identify clapperboards within frames of video and extract text from those clapperboards. The exploration consisted of:<\/p>\n

    \n
  1. Automatic metadata and insights extraction using Azure Video Analyzer for Media<\/a> (formerly known as Video Indexer).<\/li>\n
  2. Selecting and generating a representative dataset of frames for model training.<\/li>\n
  3. Augmenting images within the dataset by utilizing crop, rotate, cutout, blur, flip techniques and more.<\/li>\n
  4. A semi-supervised labeling tool to automatically group similar labels using density-based spatial clustering.<\/li>\n
  5. Training and utilizing Azure Cognitive Services Custom Vision<\/a> models to detect clapperboard images against various backdrops.<\/li>\n
  6. Clapperboard detection with trained Azure Cognitive Services Custom Vision<\/a> models<\/li>\n
  7. Training a custom model in Azure Form Recognizer<\/a> to detect and extract items such as \u201croll\u201d, \u201cscene\u201d, \u201ctake\u201d, etc. in detected clapperboard images.<\/li>\n
  8. Using Azure Cognitive Services Computer Vision Read API<\/a> (OCR) to choose the \u201cbest\u201d clapperboard frame (clapperboard image with the most detectable text) for each action event.<\/li>\n
  9. Using a trained custom model in Azure Form Recognizer<\/a> to extract key value pairs (such as \u201cscene\u201d, \u201croll\u201d, \u201ctake\u201d, etc.) from detected clapperboard images.<\/li>\n<\/ol>\n

    Audio Exploration:<\/h4>\n

    <\/p>\n

    Here we had to accurately detect the spoken word \u201ccut\u201d. The exploration consisted of:<\/p>\n