Object Detection in ML.NET Model Builder

ML.NET is an open-source, cross-platform machine learning framework for .NET developers that enables integration of custom machine learning models into .NET apps.

A new version of Model Builder is now available which includes support for object detection using your local CPU or GPU.

Download or update to the latest version of Model Builder and try it out!

What is object detection?

Object detection is a computer vision problem. While closely related to image classification, object detection performs image classification at a more granular scale. Object detection both locates and categorizes entities within images. Use object detection when images contain multiple objects of different types.

Picture of pug with bounding box

Some use cases for object detection include:

Workplace safety
Object counting
Activity recognition
Robotics
Self-driving cars

Object detection in Model Builder

We’re excited to announce you can now train object detection models in Model Builder using your local CPU or GPU.

The local object detection scenario in Model Builder is powered by the Object Detection API in ML.NET.

Similar to other deep learning APIs in ML.NET like Text Classification and Sentence Similarity, the Object Detection API is a high-level abstraction where you only need to provide your data and a few parameters to help guide the model training process.

Under the covers, the Object Detection API leverages some of the latest techniques from Microsoft Research and is backed by a Transformer-based neural network architecture built with TorchSharp. For more details on the underlying model, see the Searching the Space of Vision Transformer paper.

We’ve made the process of getting started with the Object Detection API even easier by integrating it into tools like Model Builder.

If you don’t have a GPU or need more powerful compute than the one available on your device, you also have the option of using Azure.

Get started with object detection locally

Download or update to the latest version of Model Builder

Start Model Builder by right-clicking a .NET project and choose Add > Machine Learning Model
Choose the Object Detection scenario.
Choose one of the local environments or Azure. We strongly recommend using a GPU if you have one. For more details on using GPUs in Model Builder, see the Model Builder GPU guidance.
Add your data. Model Builder supports both the VoTT and COCO formats.
Train your model. This may take a few minutes depending on how much data you have and the hardware you’re using to train your model.
Try out a few images to see whether the model is working as expected.

At this point, you can move to the next steps and use your model inside of your .NET applications.

For more details, check out the Model Builder object detection tutorial

Acknowledgements

We’d like to thank our Microsoft Research and TorchSharp partners for helping us deliver these new scenarios and capabilities in ML.NET.

Thanks to Ambrosio Blanco, Yaoyao Chang, Niklas Gustaffson, Han Hu, Yangyu Huang, Hang Li, Qingtao Li, Zhe Liu, Houwen Peng, Chi Wang, Quifeng Yin, Yilei Yang, and many others.