{"id":33324,"date":"2017-09-19T11:42:55","date_gmt":"2017-09-19T18:42:55","guid":{"rendered":"https:\/\/blog.xamarin.com\/?p=33324"},"modified":"2019-04-04T07:48:36","modified_gmt":"2019-04-04T14:48:36","slug":"augment-reality-xamarin-ios-11","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/xamarin\/augment-reality-xamarin-ios-11\/","title":{"rendered":"Augment Reality and ARKit with Xamarin and iOS 11"},"content":{"rendered":"<p>\t\t\t\tOne of the showcase features of iOS 11 is ARKit, an augmented-reality mode available on devices powered by A9 and newer chips (basically, 6S and better iPhones, 2017 iPads, and iPad Pros). With ARKit, users hold up the device and view a composite of the video feed and computer-generated imagery (either 2D SpriteKit or 3D SceneKit) that appears &#8220;attached&#8221; to horizontal surfaces in the real world.<\/p>\n<h2>Ultra-brief overview of ARKit<\/h2>\n<p>The lion&#8217;s share of work in ARKit is done behind-the-scenes on the device. Basically, a <code>ARSession<\/code> object combines device motion data, the camera&#8217;s physical characteristics (focal length, pixel size, etc.), and computational geometry to detect trackable &#8220;feature points&#8221; in the input video stream, locates them relative to a fixed world coordinate system, and creates <code>ARAnchor<\/code> objects that can be used to bridge between real-world and computer-generated imagery.<\/p>\n<p>There are limitations. As mentioned previously, ARKit leaves behind older iOS devices. Also, ARKit can only detect horizontal surfaces (<code>ARPlaneDetection<\/code> is an enumeration that only defines <code>Horizontal<\/code>, but I have to believe that the limitation is due to some quirky behavior that Apple will fix sooner rather than later). Finally, the computer imagery in an AR scene is rendered above the video and appears above real-world objects that would occlude it.<\/p>\n<p>The five critical concepts in ARKit are:<\/p>\n<ul>\n<li>Everything is defined in terms of the world coordinate system, which is initialized soon after the <code>ARSession<\/code> begins running.<\/li>\n<li>Image-processing finds high-contrast points in the real world that are stable from frame-to-frame. These &#8220;feature points&#8221; are intermediate results that are available to the developer, but mostly inform the system&#8217;s detection of <code>ARPlaneAnchor<\/code> objects. A good number of feature points are necessary for quite a few frames before ARKit detects a plane. Not surprisingly, bright lighting and textured surfaces seem to generate many more trackable feature points than evenly-illuminated matte surfaces.<\/li>\n<li>Planes are created, removed, and coalesced as the image-processing continues. Their extent shifts, expands, and shrinks, depending on the world-tracking.<\/li>\n<li>Computer imagery is anchored to these planes and rendered on the screen as if at that location and orientation in the real world.<\/li>\n<li>Because AR processing occurs 60 times per second, for optimal performance you need to be careful about memory and disposing of no-longer-needed resources. Xamarin\u2019s <a href=\"https:\/\/www.xamarin.com\/profiler\">Profiler<\/a> is your friend!<\/li>\n<\/ul>\n<p>The following two images show these concepts. The first image shows the axes of the world origin &#8220;floating in space&#8221;, the current feature points as a set of yellow dots, and a small red cube where ARKit placed a <code>ARPlaneAnchor<\/code>.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/ARKit1.png\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-33331\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/ARKit1.png\" alt=\"\" width=\"447\" height=\"600\" \/><\/a><\/p>\n<p>The second image shows a conceptual view of the world coordinate system, camera \/ iOS device, and anchor, as well as showing the plane defined by the <code>ARPlaneAnchor<\/code> and a piece of 3D SceneKit geometry placed relative to the world coordinate system and the plane.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/ARKit2.png\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-33332\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/ARKit2.png\" alt=\"\" width=\"600\" height=\"407\" \/><\/a><\/p>\n<h3>The Simplest ARKit That Could Work<\/h3>\n<p>Developers actually have complete control over rendering in ARKit but many scenarios will use the predefined <code>ARSCNView<\/code> and <code>ARSKView<\/code> that provide augmented-reality content based on SceneKit (3D) and SpriteKit (2D) respectively.<\/p>\n<p>The following ARKit program allows you to place a piece of SceneKit geometry (a cube) so that it appears &#8220;attached to&#8221; a recognized horizontal plane. Before that happens, ARKit has to be initialized and run long enough to recognize a horizontal plane.<\/p>\n<p>I initialize ARKit in two methods: <code>ViewDidLoad<\/code> and <code>ViewWillAppear<\/code>. The first to run is <code>ViewDidLoad<\/code>:<\/p>\n<pre class=\"lang:c# decode:true\">public class ARKitController : UIViewController\n{\t\t\n\tARSCNView scnView; \t\t\n\tpublic ARKitController() : base(){ }\t\n\t\n\tpublic override bool ShouldAutorotate() =&gt; true;\t\t\n\tpublic override void ViewDidLoad()\t\n\t{\t\t\t\n\t\tbase.ViewDidLoad();\t\t\t\n\t\tscnView = new ARSCNView()\t\t\n\t\t{\t\t\t\t\n\t\t\tFrame = this.View.Frame,\t\t\t\n\t\t\tDelegate = new ARDelegate(),\t\t\t\t\n\t\t\tDebugOptions = ARSCNDebugOptions.ShowFeaturePoints | ARSCNDebugOptions.ShowWorldOrigin,\t\t\t\t\n\t\t\tUserInteractionEnabled = true\t\t\t\n\t\t};\t\t\n\t\t\n\t\tthis.View.AddSubview(scnView);\t\n\t}\n\/\/... code continues ... \n<\/pre>\n<p>This is a typical iOS <code>UIViewController<\/code> subclass. It creates a new <code>ARSCNView<\/code> (the easy-to-use route towards SceneKit 3D geometry), sets its delegate-object to an instance of a class I wrote called <code>ARDelegate<\/code> (discussed later), turns on some debug visualizations, enables the view to respond to touches, and adds the <code>ARSCNView<\/code> to the view hierarchy.<\/p>\n<p>The second part of initialization occurs during <code>ViewWillAppear<\/code> (it may be that this could be done during <code>ViewDidLoad<\/code> but I am always a little leery of the highly stateful view-initialization process in iOS):<\/p>\n<pre class=\"lang:c# decode:true\">public override void ViewWillAppear(bool animated)\n{\t\n\tbase.ViewWillAppear(animated);\t\n\t\n\t\/\/ Configure ARKit \t\n\tvar config = new ARWorldTrackingConfiguration();\t\n\tconfig.PlaneDetection = ARPlaneDetection.Horizontal;\n\t\t\n\t\/\/ This method is called subsequent to ViewDidLoad so we know scnView is instantiated\t\n\tscnView.Session.Run(config, ARSessionRunOptions.RemoveExistingAnchors);\n}\n<\/pre>\n<p>This simply configures the <code>ARSession<\/code> of the previously-created <code>ARSCNView<\/code> and begins its processing.<\/p>\n<p>At this point, ARKit begins background processing. On the screen, the user will see the feed from their camera and, in a second or two, the debug visualizations will begin to show the feature cloud and world-coordinates origin (similar to the screenshot image shown previously).<\/p>\n<p>At some point after that, ARKit will hopefully discover enough co-planar feature points to track a horizontal plane. When <em>that<\/em> happens, ARKit will automatically add an <code>ARPlaneAnchor<\/code> to its world-tracking. That addition triggers the <code>DidAddNode<\/code> method on the <code>ARSCNView<\/code> object&#8217;s delegate-object, which in our case is <code>ARDelegate<\/code>:<\/p>\n<pre class=\"lang:c# decode:true\">public class ARDelegate : ARSCNViewDelegate\n{\t\n\tpublic override void DidAddNode(ISCNSceneRenderer renderer, SCNNode node, ARAnchor anchor)\t\n\t{\n\t\tif (anchor != null &amp;&amp; anchor is ARPlaneAnchor)\n\t\t{\t\n\t\t\tPlaceAnchorNode(node, anchor as ARPlaneAnchor);\n\t\t}\n\t}\t\n\t\t\t\t\n\tvoid PlaceAnchorNode(SCNNode node, ARPlaneAnchor anchor)\t\n\t{\n\t\tvar plane = SCNPlane.Create(anchor.Extent.X, anchor.Extent.Z);\n\t\tplane.FirstMaterial.Diffuse.Contents = UIColor.LightGray;\n\t\t\n\t\tvar planeNode = SCNNode.FromGeometry(plane);\n\n\t\t\/\/Locate the plane at the position of the anchor\n\t\tplaneNode.Position = new SCNVector3(anchor.Extent.X, 0.0f, anchor.Extent.Z);\n\n\t\t\/\/Rotate it to lie flat\n\t\tplaneNode.Transform = SCNMatrix4.CreateRotationX((float) (Math.PI \/ 2.0));\n\t\tnode.AddChildNode(planeNode);\n\n\t\t\/\/Mark the anchor with a small red box\n\t\tvar box = new SCNBox \n\t\t{ \n\t\t\tHeight = 0.1f, \n\t\t\tWidth = 0.1f, \n\t\t\tLength = 0.1f \n\t\t};\n\t\tbox.FirstMaterial.Diffuse.ContentColor = UIColor.Red;\n\t\tvar anchorNode = new SCNNode \n\t\t{ \n\t\t\tPosition = new SCNVector3(0, 0, 0), \n\t\t\tGeometry = box \n\t\t};\n\t\tplaneNode.AddChildNode(anchorNode);\t\n\t}\t\n\t\n\tpublic override void DidUpdateNode(ISCNSceneRenderer renderer, SCNNode node, ARAnchor anchor)\t\n\t{\n\t\tif (anchor is ARPlaneAnchor)\n\t\t{\t\n\t\t\tvar planeAnchor = anchor as ARPlaneAnchor;\t\n\t\t\tSystem.Console.WriteLine($\"The (updated) extent of the anchor is [{planeAnchor.Extent.X} , {planeAnchor.Extent.Y} , {planeAnchor.Extent.Z} ]\");\n\t\t}\n\t}\n} \n<\/pre>\n<p><code>DidAddNode<\/code> is called any time a node is added to the view, but we are only interested in special processing if it&#8217;s an <code>ARPlaneAnchor<\/code>, indicating that ARKit is adding to its internal model of horizontal planes. We test for that condition and, if <code>true<\/code>, call <code>this.PlaceAnchorCube<\/code>. That method, in turn, creates some SceneKit geometry: a node that holds the plane&#8217;s geometry and is positioned in the same world coordinate as the anchor, and a small red box as a visual indicator of the <code>ARPlaneAnchor<\/code>. Note that because SceneKit uses a scene-graph architecture, the <code>anchorNode<\/code> position of [0,0,0] is relative to its parent&#8217;s position &#8212; the <code>planeNode<\/code>, whose position is based on the <code>planeAnchor<\/code>, whose position, in turn, is in world coordinates.<\/p>\n<p>Once this method is called, the user will see something essentially identical to the screenshot above.<\/p>\n<h2>Hit-Testing<\/h2>\n<p>Once at least one plane is detected and tracked, the user can place additional geometry by touching on the screen. Back in the <code>ViewController<\/code> class:<\/p>\n<pre class=\"lang:c# decode:true\">\/\/ This snippet is part of:\npublic class ARKitController : UIViewController\n{\n\t\/\/\u2026 code shown previously \u2026\n\tpublic override void TouchesBegan(NSSet touches, UIEvent evt)\n\t{\n\t\tbase.TouchesBegan(touches, evt);\n\t\tvar touch = touches.AnyObject as UITouch;\n\t\tif (touch != null)\n\t\t{\n\t\t\tvar loc = touch.LocationInView(scnView);\n\t\t\tvar worldPos = WorldPositionFromHitTest(loc);\n\t\t\tif (worldPos != null)\n\t\t\t{\n\t\t\t\tPlaceCube(worldPos.Item1);\n\t\t\t}\n\t\t}\n\t}\n\tprivate SCNVector3 PositionFromTransform(NMatrix4 xform)\n\t{\t\n\t\treturn new SCNVector3(xform.M14, xform.M24, xform.M34);\n\t}\n\n\t(SCNVector3, ARAnchor) WorldPositionFromHitTest (CGPoint pt)\n\t{\t\n\t\t\/\/Hit test against existing anchors\t\n\t\tvar hits = scnView.HitTest(pt, ARHitTestResultType.ExistingPlaneUsingExtent);\t\n\t\tif (hits != null &amp;&amp; hits.Length &gt; 0)\t\n\t\t{\t\t\n\t\t\tvar anchors = hits.Where(r =&gt; r.Anchor is ARPlaneAnchor);\t\t\n\t\t\tif (anchors.Count() &gt; 0)\t\t\n\t\t\t{\t\t\t\n\t\t\t\tvar first = anchors.First();\t\t\t\n\t\t\t\tvar pos = PositionFromTransform(first.WorldTransform);\t\t\t\n\t\t\t\treturn (pos, (ARPlaneAnchor)first.Anchor);\t\t\n\t\t\t}\t\n\t\t}\t\n\t\t\n\t\treturn null;\n\t}\n\t\t\n\tprivate SCNMaterial[] LoadMaterials()\n\t{\t\n\t\tFunc&lt;string, SCNMaterial&gt; LoadMaterial = fname =&gt;\t\n\t\t{\t\t\n\t\t\tvar mat = new SCNMaterial();\t\t\n\t\t\tmat.Diffuse.Contents = UIImage.FromFile(fname);\t\t\n\t\t\tmat.LocksAmbientWithDiffuse = true;\t\treturn mat;\t\n\t\t};\t\n\t\t\n\t\tvar a = LoadMaterial(\"msft_logo.png\");\t\n\t\tvar b = LoadMaterial(\"xamagon.png\");\t\n\t\tvar c = LoadMaterial(\"fsharp.png\"); \n\n\t\t\/\/ This demo was originally in F# :-) \t\n\t\treturn new[] { a, b, a, b, c, c };\n\t}\n\t\t\n\tSCNNode PlaceCube(SCNVector3 pos)\n\t{\t\n\t\tvar box = new SCNBox \n\t\t{ \n\t\t\tWidth = 0.10f, \n\t\t\tHeight = 0.10f, \n\t\t\tLength = 0.10f \n\t\t};\t\n\t\t\n\t\tvar cubeNode = new SCNNode \n\t\t{ \n\t\t\tPosition = pos, \n\t\t\tGeometry = box \n\t\t};\t\n\t\t\n\t\tcubeNode.Geometry.Materials = LoadMaterials();\t\n\t\tscnView.Scene.RootNode.AddChildNode(cubeNode);\t\n\t\treturn cubeNode;\n\t}\n}\t\n<\/pre>\n<p>Skip to the <code>TouchesBegan<\/code> method and you can see that what we do is pretty straightforward at a high level: we grab the position of the touch and then perform a hit test for horizontal planes. If that hit-test is successful, we return the position on the first such plane as a tuple of type <code>(SCNVector3, ARPlaneAnchor)<\/code>. The hit-testing is done using the built-in <code>ARSceneView.HitTest(SCNVector3, ARHitTestResultType)<\/code> method, which projects a ray outward into the augmented-reality &#8220;world&#8221; and returns an array containing anchors on any of the planes that it intersects, ordered nearest-to-furthest. If that array is not empty, we grab the first and return its position as an <code>SCNVector3<\/code> (which we extract from the appropriate components of the anchor&#8217;s <code>NMatrix4<\/code>\u00a0matrix). (Historical note: During the iOS 11 beta period, the type used for these matrices switched between row-major and column-major in the Xamarin bindings. If you review code written during the beta period, rotations and translations may appear transposed.)<\/p>\n<p>The <code>PlaceCube<\/code> method just creates a box 10cm on a side and places it in the augmented-reality &#8220;world&#8221; at <code>pos<\/code>, whose value is the <code>SCNVector3<\/code> returned by <code>WorldPositionFromHitTest<\/code> as mentioned above.<\/p>\n<p>The result is something like:<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/ARKit3.jpeg\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-33333\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/ARKit3.jpeg\" alt=\"\" width=\"338\" height=\"600\" \/><\/a><\/p>\n<h2>Learn More<\/h2>\n<p>The <a href=\"https:\/\/twitter.com\/hashtag\/madewitharkit?src=hash\">#MadeWithARKit<\/a> hashtag on Twitter has been a great source of inspiration this Summer, with people demo\u2019ing such great concepts as virtual portals, sound nodes located in space, and the combination of ARKit with video filters.<\/p>\n<p>After the iPhone X launch announcement, Apple revealed some new APIs relating to face detection and mapping, including an <code>ARFaceAnchor<\/code> and <code>ARFaceGeometry<\/code>.<\/p>\n<p>All the code for this sample is available at <a href=\"https:\/\/github.com\/lobrien\/ARKit_Csharp\">https:\/\/github.com\/lobrien\/ARKit_Csharp<\/a>. Pull-requests and questions welcome! Make sure to also\u00a0check out the <a href=\"https:\/\/developer.xamarin.com\/guides\/ios\/platform_features\/introduction-to-ios11\/\">Introduction to iOS 11 guide<\/a> in the Xamarin Developer Center.<\/p>\n<p><a href=\"https:\/\/forums.xamarin.com\/103549\/\">Discuss this post in the Xamarin Forums<\/a>\t\t<\/p>\n","protected":false},"excerpt":{"rendered":"<p>One of the showcase features of iOS 11 is ARKit, an augmented-reality mode available on devices powered by A9 and newer chips (basically, 6S and better iPhones, 2017 iPads, and iPad Pros). With ARKit, users hold up the device and view a composite of the video feed and computer-generated imagery (either 2D SpriteKit or 3D [&hellip;]<\/p>\n","protected":false},"author":556,"featured_media":33333,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2],"tags":[6,4],"class_list":["post-33324","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-developers","tag-ios","tag-xamarin-platform"],"acf":[],"blog_post_summary":"<p>One of the showcase features of iOS 11 is ARKit, an augmented-reality mode available on devices powered by A9 and newer chips (basically, 6S and better iPhones, 2017 iPads, and iPad Pros). With ARKit, users hold up the device and view a composite of the video feed and computer-generated imagery (either 2D SpriteKit or 3D [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/posts\/33324","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/users\/556"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/comments?post=33324"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/posts\/33324\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/media?parent=33324"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/categories?post=33324"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/tags?post=33324"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}