{"id":35666,"date":"2018-02-20T14:36:35","date_gmt":"2018-02-20T19:36:35","guid":{"rendered":"https:\/\/blog.xamarin.com\/?p=35666"},"modified":"2019-04-04T15:37:20","modified_gmt":"2019-04-04T22:37:20","slug":"coreml-azure-create-simple-xamarin-ios-apps","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/xamarin\/coreml-azure-create-simple-xamarin-ios-apps\/","title":{"rendered":"Use CoreML And Azure To Create Simple Xamarin.iOS Apps"},"content":{"rendered":"

Last year Apple released a device-optimized machine learning framework called Core ML to make it as easy as possible to integrate machine learning and artificial intelligence services into your iOS and Mac apps. CoreML is a blessing for developers who lack extensive knowledge of AI or machine learning, because getting started only requires referencing a pre-trained model in a project and adding a few lines of code.<\/p>\n

In this blog post, we’ll discuss how to create a simple iOS app to identify Links, Hylian Shield, and a Master Sword using CoreML and Azure.<\/p>\n

Getting started<\/h2>\n

While consuming pre-trained models is easiest, we often don\u2019t have large data sets available to train the model. To solve this problem, Microsoft created the Custom Vision Service<\/a>, which generates custom machine learning models with just five images in only a few minutes. These models can be exported into CoreML models and consumed in iOS and Mac apps.<\/p>\n

Let\u2019s start by creating a project with Custom Vision Service. Make sure you select a compact<\/code> model under Domain. The Compact domains can be exported to Core ML or Tensorflow compatible models to run on the devices locally:<\/p>\n

\"\"<\/p>\n

Next, a trained model will be ready for use in three easy steps:<\/p>\n

    \n
  1. Upload training images.<\/li>\n
  2. Tag the images.<\/li>\n
  3. Click Train.<\/li>\n<\/ol>\n

    It only takes few minutes to complete the training, and then you can download the model to use in your app.<\/p>\n

    \"\"<\/p>\n

    For Your iOS App<\/h2>\n

    Now that we have our CoreML model set up, it\u2019s time to add it to your iOS app project.<\/p>\n

    Add Model to the project<\/h3>\n

    Create a new single view iOS app by going to File \u2192 New Project \u2192 iOS \u2192 Single View in Visual Studio 2017. Add the downloaded model (file with .mlmodel extension) to the Resources<\/code> folder, and make sure the BuildAction<\/code> is set to BundleResource<\/code>.<\/p>\n

    Initialize Vision Model<\/h3>\n

    Before using the model, we need to load it in our app and initialize it with a request handler, where we can read ML\u2019s observations:<\/p>\n

    \r\n    var modelPath = NSBundle.MainBundle.GetUrlForResource(\"Link\", \"mlmodel\");\r\n    var compiledPath = MLModel.CompileModel(modelPath, out NSError compileError);\r\n    var mlModel = MLModel.Create(compiledPath, out NSError createError);\r\n\r\n    var model = VNCoreMLModel.FromMLModel(mlModel, out NSError mlError);\r\n\r\n    \/\/ Initialise classification request and Handler\r\n    var classificationRequest = new VNCoreMLRequest(model, HandleClassificationRequest);\r\n<\/code><\/pre>\n
    This code compiles the downloaded mlmodel<\/code> on the device, making it easy to update models without updating the app:<\/p>\n
    \"\"<\/p>\n
    Request classification<\/h3>\n
    Now we need to identify the objects. The following code does this when we take a photo on device:<\/p>\n
    \r\n    var requestHandler = new VNImageRequestHandler(imageNSData, new VNImageOptions());\r\n    requestHandler.Perform(classificationRequest, out NSError error);\r\n<\/code><\/pre>\n
    Read the observations<\/h3>\n
    As soon as something is detected, the classification request will invoke HandleClassificationRequest<\/code> method, where we can read the observations from the request<\/code> parameter using GetResults<\/code> method:<\/p>\n
    \r\n    void HandleClassificationRequest(VNRequest request, NSError error)\r\n    {\r\n        var observations = request.GetResults();\r\n        var best = observations?[0];\r\n\r\n        Debug.WriteLine($\"{best.Identifier}, Confidence: {best.Confidence:P0}\");\r\n    }\r\n<\/code><\/pre>\n
    The best possible classification will be available at the 0<\/code> index, with an identification label, and confidence value in the range 0..1<\/code>, where 0<\/code> means no confidence that the observation is accurate and 1<\/code> means certainty.\n \n