{"id":33978,"date":"2017-10-30T12:50:24","date_gmt":"2017-10-30T19:50:24","guid":{"rendered":"https:\/\/blog.xamarin.com\/?p=33978"},"modified":"2019-04-04T07:48:08","modified_gmt":"2019-04-04T14:48:08","slug":"augmented-reality-xamarin-android-arcore","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/xamarin\/augmented-reality-xamarin-android-arcore\/","title":{"rendered":"Augmented Reality in Xamarin.Android with ARCore"},"content":{"rendered":"<p>\t\t\t\t<img decoding=\"async\" class=\"alignright wp-image-33982\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/arcore-xamarin.jpg\" alt=\"\" width=\"300\" height=\"479\" \/>Now that you&#8217;ve had a chance to augment reality in your <a href=\"https:\/\/blog.xamarin.com\/augment-reality-xamarin-ios-11\/\">Xamarin iOS apps with ARKit<\/a>, it&#8217;s time to explore Google&#8217;s take on AR in your Xamarin Android apps.<\/p>\n<p>The new <a href=\"https:\/\/developers.google.com\/ar\/discover\/\">ARCore SDK<\/a> provides APIs for Augmented Reality features, such as motion tracking, plane detection, and light estimation. These are the building blocks you will use to add AR experiences to your Android apps.<\/p>\n<h2>Getting Started with ARCore<\/h2>\n<p>ARCore is currently only available on select devices such as the Google Pixel, Google Pixel 2, and the Samsung Galaxy S8.<\/p>\n<p>In order to use ARCore, you need to <a href=\"https:\/\/developers.google.com\/ar\/develop\/java\/getting-started#prepare-device\">prepare your device<\/a> by downloading and installing <a href=\"https:\/\/github.com\/google-ar\/arcore-android-sdk\/releases\/download\/sdk-preview\/arcore-preview.apk\"><strong><em>arcore-preview.apk<\/em><\/strong><\/a>.<\/p>\n<p>After you set up your device for ARCore development, you need to install the ARCore prerelease <a href=\"https:\/\/www.nuget.org\/packages\/Xamarin.Google.ARCore\/\">NuGet package<\/a>.<\/p>\n<h2>ARCore API Basics<\/h2>\n<p>To help you detect surfaces to place objects on and calculate their location in space relative to the camera, ARCore uses a few basic types.<\/p>\n<ul>\n<li><strong>Session Object<\/strong>: Your main point of interaction with ARCore. It will help manage the AR state by keeping track of any anchors you add, surfaces the engine has detected, and current snapshots of the device.<\/li>\n<li><strong>Plane<\/strong>: A surface the SDK has detected, onto which you can place an <code>Anchor<\/code>\u00a0to describe a fixed real-world location of an object (including its orientation). Currently, surfaces facing upward and downward can be detected separately (think floor and ceiling).<\/li>\n<li><strong>Session<\/strong>: The current snapshot of the AR state when you call <code>.Update()<\/code>,\u00a0which returns a <code>Frame Object<\/code>.<\/li>\n<li><strong>Frame<\/strong>: A convenient <code>HitTest(..)<\/code>\u00a0method, which can help determine if tapped coordinates on the display intersect with any planes, for example. Each frame also contains information about the camera&#8217;s orientation and relation to the real world and helps compute projection matrices for displaying visual representations to the user.<\/li>\n<li>Another interesting feature of the SDK is the ability to obtain a <code>LightEstimate<\/code>\u00a0from a given frame. This estimate includes the <code>PixelIntensity<\/code>\u00a0of the camera view.<\/li>\n<\/ul>\n<h2>Basic Walkthrough<\/h2>\n<p>We&#8217;ve ported the <a href=\"https:\/\/github.com\/xamarin\/XamarinComponents\/tree\/master\/Android\/ARCore\/samples\">HelloAR sample<\/a> to Xamarin, and you can go check it out on GitHub! Now let&#8217;s walk through a few of the basic things going on in this sample. <\/p>\n<p>First, in your activity you need to create a session in <code>OnCreate<\/code>\u00a0and make sure ARCore is supported on the device at runtime:<\/p>\n<pre class=\"lang:c# decode:true\">var config = Config.CreateDefaultConfig();\nsession = new Session(this);\n\n\/\/ Make sure ARCore is supported on this device\nif (!session.IsSupported(config)) {\n    Toast.MakeText(this, \"ARCore unsupported!\", ToastLength.Long).Show();\n    Finish();\n}<\/pre>\n<p><br><\/p>\n<p>Remember, you also need to request <em>Android.Manifest.Permission.Camera<\/em>\u00a0permissions to display the live camera feed \/ augmented reality view to the user. In the &#8220;HelloAR&#8221; sample, we use a <code>GLSurfaceView<\/code>\u00a0to render camera and augmentations to the user. Make sure you set up your GL Surface or look at how it&#8217;s done in the sample code.<\/p>\n<p>With a session running, we can obtain a snapshot of the AR system state in our GL surface&#8217;s <code>OnDrawFrame<\/code>\u00a0implementation. After we check to ensure the frame is in a <code>Tracking<\/code>\u00a0 state, we can check for any hit results and, assuming they intersect a plane, add an anchor to the plane.<\/p>\n<pre class=\"lang:c# decode:true\">\/\/ See the PlaneAttachment class from the sample\n\/\/ This helps associate Anchors with Planes they are attached to\nList&lt;PlaneAttachment&gt; planeAttachments = new List&lt;PlaneAttachment&gt;();\nvoid OnDrawFrame (IGL10 gl)\n{\n  \u00a0 var frame = session.Update();\n\n\u00a0   \/\/ You could keep track of taps by queueing up\n    \/\/ MotionEvent's from a tap gesture recognizer\n    var tap = motionEventsQueue.Dequeue();\n\n    \/\/ Make sure we've got a tap and are in a tracking state for our frame\n    if (tap != null &amp;&amp; frame.GetTrackingState() == Frame.TrackingState.Tracking) {\n        \/\/ Look at each hittest result\n        foreach (var hit in frame.HitTest(tap)) {\n            \/\/ We could get PointCloudsHitResult as well, check for Plane\n            var planeHit = hit as PlaneHitResult;\n            if (planeHit != null &amp;&amp; planeHit.IsHitInPolygon) {\n                \/\/ Create a new anchor\n                var anchor = session.AddAnchor(hit.HitPose);\n                \/\/ Keep track of our anchors and the planes they are attached to\n                planeAttachments.Add(new PlaneAttachment(planeHit, anchor))\n            }\n        }\n    }\n}<\/pre>\n<p>We also want to render the various objects in our scene in our drawing method. The HelloAR sample has various renderers to do the heavy OpenGL lifting and achieve this based on the projections calculated from the frame:<\/p>\n<pre class=\"lang:c# decode:true \">\/\/ Get projection matrix.\nfloat[] projectionMatrix = new float[16];\nsession.GetProjectionMatrix(projectionMatrix, 0, 0.1f, 100.0f);\n\n\/\/ Get camera matrix and draw.\nfloat[] viewMatrix = new float[16];\nframe.GetViewMatrix(viewMatrix, 0);\n\n\/\/ Draw the detected planes\nplaneRenderer.DrawPlanes(session.AllPlanes, frame.Pose, projectionMatrix);\n\n\/\/ Get lighting from avg intensity of the image\nvar lightIntensity = frame.LightEstimate.PixelIntensity;\n\n\/\/ Draw all of our anchors attached to planes\nfloat scaleFactor = 1.0f;\nfloat[] anchorMatrix = new float[16];\n\nforeach (var planeAttachment in planeAttachments) {\n    \/\/ Only draw attachments currently tracking\n    if (!planeAttachment.IsTracking)\n        continue;\n\n    \/\/ Get the current combined pose of an Anchor and Plane in world space\n    planeAttachment.GetPose().ToMatrix(anchorMatrix, 0);\n\n    \/\/ Update and draw the model\n    objectRenderer.UpdateModelMatrix(anchorMatrix, scaleFactor);\n    objectRenderer.Draw(viewMatrix, projectionMatrix, lightIntensity);\n}<\/pre>\n<p><br><\/p>\n<p>After dissecting the sample, you can see the actual ARCore code is relatively straightforward and most of the sample code is about the OpenGL rendering.\u00a0 <\/p>\n<p>Again, be sure to check out the <a href=\"https:\/\/github.com\/xamarin\/XamarinComponents\/tree\/master\/Android\/ARCore\/samples\">HelloAR sample<\/a> in its entirety on GitHub! We look forward to seeing what Augmented Reality experiences you create with ARCore in your Xamarin Android apps. <\/p>\n<p><A href=\"https:\/\/forums.xamarin.com\/\">Discuss this post on the forums!<\/a>\t\t<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Now that you&#8217;ve had a chance to augment reality in your Xamarin iOS apps with ARKit, it&#8217;s time to explore Google&#8217;s take on AR in your Xamarin Android apps. The new ARCore SDK provides APIs for Augmented Reality features, such as motion tracking, plane detection, and light estimation. These are the building blocks you will [&hellip;]<\/p>\n","protected":false},"author":560,"featured_media":33982,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2],"tags":[5,4],"class_list":["post-33978","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-developers","tag-android","tag-xamarin-platform"],"acf":[],"blog_post_summary":"<p>Now that you&#8217;ve had a chance to augment reality in your Xamarin iOS apps with ARKit, it&#8217;s time to explore Google&#8217;s take on AR in your Xamarin Android apps. The new ARCore SDK provides APIs for Augmented Reality features, such as motion tracking, plane detection, and light estimation. These are the building blocks you will [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/posts\/33978","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/users\/560"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/comments?post=33978"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/posts\/33978\/revisions"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/media?parent=33978"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/categories?post=33978"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/tags?post=33978"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}