![\"Creating](\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/lobrien_point_cloud.jpeg\")
<\/p>\n\t\t\t\tIt’s summertime, which for Xamarin developers means new iOS betas to explore and learn. ARKit, which debuted last year in iOS 11, has matured over the past year and in iOS 12 is much more flexible than it was previously.<\/p>\n
If you haven’t looked at ARKit in a year, the truly big change came from a point-release in the spring, when Apple added the capability to detect vertical and horizontal planes. Detected planes are the major junction between computer-generated geometry and the real world. Most often, you’ll position your computer imagery relative to such planes.<\/p>\n
<\/p>\n
These planes are built upon a lower-level feature; a “point cloud” generated by the system using “visual inertial odometry<\/a>.”<\/p>\n Fun fact:<\/strong> the ‘inertial’ part of that includes exploiting the slight shifts inevitable from holding the device in your hand, so ARKit doesn’t work as well if you put the device on a tripod.<\/em><\/p>\n This point cloud is available to developers in the Visualizing the point cloud efficiently can be a tricky problem. Iterating over the points, then creating and placing a new SceneKit node for each point would kill the frame rate. Alternatively, if done asynchronously there would be a lag. The sample maintains performance with a three-part strategy:<\/p>\n Another tricky challenge relates to using gestures to manipulate AR objects. The app allows the user to drag, rotate, scale, and extrude sides of the bounding box that surrounds the object being scanned. First, the gestures are threshold-aware and only activate after the user has moved a certain amount of pixels — the code in the Gesture_Recognizers<\/strong> subdirectory is highly reusable and makes a nice addition to your catalog.<\/p>\n Even more interesting is the challenge of dragging the bounding box along a real-world plane. Such as the surface on which the scanned object sits. In order to constrain the drag to a specific axis (say the X-axis), the easiest way is to construct a new plane that is:<\/p>\n This situation for dragging along the X-axis is shown in the following illustration. To drag along the red X-axis, while viewing the bounding box from the vector shown by the yellow arrow, we need to construct the yellow plane:<\/p>\n The constructor for Another option would be to extract the 4 component values of each of the 4 vectors and use the You can find the code above in the sample at This tends to be a common source of difficulties when dealing with Swift 3D graphics! Most of the Swift sample code uses row-major transform matrices. Both column- and row- major interpretations are valid<\/a>. Column-major is more common in mathematical texts while row-major may be a better fit for optimizing performance. In vector-matrix multiplication, for instance, the n<\/em>th element in the vector is multiplied by the n<\/em>th row of the matrix. Having the row data adjacent in memory can be a performance boost with\u00a0SIMD<\/a> operations or from cache issues with large matrices. It has been said that “column-major is developer-friendly, while row-major is hardware-friendly.”<\/p>\n In Swift example code, most transforms are based on SIMD such as This sample demonstrates several other techniques helpful in more advanced Augmented Reality applications. Techniques such as various forms of hit-testing, serializing reference objects, and leveraging non-visible SceneKit objects for reference positioning. In addition, it demonstrates an interesting nested state-machine architecture that uses ARFrame.RawFeaturePoints<\/code><\/a> property. The\u00a0Scanning and Detecting 3D Objects<\/a>\u00a0sample shows how the point cloud can be used to create and save an ARReferenceObject<\/code><\/a> for use in other ARKit apps. For instance, you could create an ARReferenceObject<\/code> of a sculpture for use in a museum app or an engine part for use in a DIY car-repair app.<\/p>\nThree-Part Performance Strategy<\/h3>\n
\n
unsafe<\/code> code to pin the data in place and interpret the data as a raw buffer of bytes.<\/li>\nSCNGeometrySource<\/code><\/a> and creating a “template” SCNGeometryElement<\/code><\/a> object.<\/li>\nSCNGeometry.Create(SCNGeometrySource[], SCNGeometryElement[])<\/code><\/a><\/li>\n<\/ul>\ninternal static SCNGeometry CreateVisualization(NVector3[] points, UIColor color, float size)\r\n{\r\n if (points.Length == 0)\r\n {\r\n return null;\r\n }\r\n\r\n unsafe\r\n {\r\n var stride = sizeof(float) * 3;\r\n\r\n \/\/ Pin the data down so that it doesn't move\r\n fixed (NVector3* pPoints = &points[0])\r\n {\r\n \/\/ Important: Don't unpin until after `SCNGeometry.Create`, because geometry creation is lazy\r\n \r\n \/\/ Grab a pointer to the data and treat it as a byte buffer of the appropriate length\r\n var intPtr = new IntPtr(pPoints);\r\n var pointData = NSData.FromBytes(intPtr, (System.nuint) (stride * points.Length));\r\n\r\n \/\/ Create a geometry source (factory) configured properly for the data (3 vertices)\r\n var source = SCNGeometrySource.FromData(\r\n pointData,\r\n SCNGeometrySourceSemantics.Vertex,\r\n points.Length,\r\n true,\r\n 3,\r\n sizeof(float),\r\n 0,\r\n stride\r\n );\r\n\r\n \/\/ Create geometry element\r\n \/\/ The null and bytesPerElement = 0 look odd, but this is just a template object\r\n var template = SCNGeometryElement.FromData(null, SCNGeometryPrimitiveType.Point, points.Length, 0);\r\n template.PointSize = 0.001F;\r\n template.MinimumPointScreenSpaceRadius = size;\r\n template.MaximumPointScreenSpaceRadius = size;\r\n\r\n \/\/ Stitch the data (source) together with the template to create the new object\r\n var pointsGeometry = SCNGeometry.Create(new[] { source }, new[] { template });\r\n pointsGeometry.Materials = new[] { Utilities.Material(color) };\r\n return pointsGeometry;\r\n }\r\n }\r\n}<\/pre>\nAdvanced iOS Gestures<\/h2>\n
Dragging<\/h3>\n
\n
![\"Illustration](\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/lobrien_drag_plane.png\")
<\/p>\n\n
\n
camRayToOrigin = Origin - cameraPos<\/code><\/li>\nxVector = new SCNVector3(1, 0, 0)<\/code><\/li>\nSCNVector3.Cross(xVector, camRayToOrigin)<\/code><\/li>\nSCNVector3.Cross(xVector, zVector)<\/code><\/li>\n![\"A](\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/lobrien_column_major.png\")
\nThis transform can be used to map the position of the user’s moving finger to slide the bounding box along the chosen drag axis (in this case, the X-axis)<\/p>\nContructing the Proper Transform<\/h3>\n
NMatrix4<\/code><\/a> is NMatrix4(Vector4 row0, Vector4 row1, Vector4 row2, Vector4 row3)<\/code>, which takes as its argument the values of the rows<\/em>, not the columns. So the code to construct the proper transform is:<\/p>\nvar transformBasedOnColumnVectors = Utilities.NMatrix4Create(new[] {\r\n Utilities.SCNVector4Create(xVector, 0),\r\n Utilities.SCNVector4Create(yVector, 0),\r\n Utilities.SCNVector4Create(zVector, 0),\r\n Utilities.SCNVector4Create(Origin, 1)\r\n });\r\n\/\/ But we created it with row-vectors, so we need to:\r\ntransformBasedOnColumnVectors.Transpose();<\/pre>\nNMatrix4<\/code> constructor that takes\u00a0its arguments using the\u00a0M{row}{col}<\/code> notation.<\/p>\nRay.DragPlaneTransform(SCNVector3)<\/code>. The transform shown above is based on column-vectors and column-major matrices, that is:<\/p>\n![\"Multiplying](\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/lobrien_vec_mat_mult.png\")
<\/p>\nColumn- and Row-<\/h3>\n
Transpose Your Matrix<\/h3>\n
simd.float4x4<\/code>. These matrices are transposed compared to those based on column-vectors. Thus, in much Swift code, you’d find the translation component in M41, M42, and M43, rather than M14, M24, and M34. And of course, the shear and rotation components would be transposed, as well. If you have a situation where your SceneKit geometry nodes jitter, fly away, or are “stuck to the origin”, you probably need to Transpose()<\/code><\/a> your matrix.<\/p>\nNSNotificationCenter<\/code><\/a> to coordinate transitions.<\/p>\nSources and Documentation<\/h2>\n
\n