{"id":1013,"date":"2020-10-29T11:43:22","date_gmt":"2020-10-29T18:43:22","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/surface-duo\/?p=1013"},"modified":"2020-10-29T11:43:22","modified_gmt":"2020-10-29T18:43:22","slug":"dual-screen-camera-sample","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/surface-duo\/dual-screen-camera-sample\/","title":{"rendered":"Working with the camera on Microsoft Surface Duo"},"content":{"rendered":"<p>\n  Hi Android developers!\n<\/p>\n<p>\n  While the Surface Duo runs Android apps just like any other Android device, there are some differences in the <a href=\"https:\/\/www.microsoft.com\/surface\/devices\/surface-duo?activetab=techSpecs\">hardware<\/a>:\n<\/p>\n<ul>\n<li>\n    There is one camera, which can behave as either front-facing or rear-facing depending on how the device is folded. Tapping the \u201cswap cameras\u201d button will change how the image is mirrored to correctly render \u201cselfies\u201d.\n  <\/li>\n<li>\n    There are two screens, so your camera app might be running at the same time as another app. Because the two apps might have different orientation requirements, your camera view might be rotated and letterboxed by the operating system. \n  <\/li>\n<\/ul>\n<p>\n  In this post, we will review some dual-screen-specific camera behaviors and present some different options for enhancing camera capture in your apps to work even better on the Surface Duo. The <a href=\"https:\/\/github.com\/microsoft\/surface-duo-sdk-samples\/tree\/master\/Camera\">complete source code<\/a> is available for a Camera sample app, shown here:\n<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2020\/10\/camera-sample-rose-portrait-500.png\"><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2020\/10\/camera-sample-rose-portrait-500.png\" alt=\"Surface Duo camera sample\" width=\"500\" height=\"387\" class=\"alignnone size-full wp-image-1018\" srcset=\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2020\/10\/camera-sample-rose-portrait-500.png 500w, https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2020\/10\/camera-sample-rose-portrait-500-300x232.png 300w\" sizes=\"(max-width: 500px) 100vw, 500px\" \/><\/a><br\/><em>Figure 1: Camera sample app<\/em>\n<\/p>\n<h2>Working with portrait orientation lock<\/h2>\n<p>\n  Many camera apps lock orientation \u2013 especially social media apps, which often lock the orientation to portrait mode. When your app is running on a single screen and is portrait locked, users will typically have the same experience as any other Android device.\n<\/p>\n<p>\n  There is one new scenario for orientation-locked apps on the Surface Duo \u2013 remaining readable when the device is rotated. For example, when an app is portrait-locked, on a single screen, and the device is in double-landscape mode, the other screen will likely have a landscape-oriented app (or launcher) showing. Orientation may be locked by an entry in the <strong>AndroidManifest.xml<\/strong> file:\n<\/p>\n<pre>android:screenOrientation=\"portrait\"<\/pre>\n<p>\n  or in code:\n<\/p>\n<pre>\r\nsetRequestedOrientation(Configuration.ORIENTATION_PORTRAIT); \/\/ Java\r\n<\/pre>\n<p>\n  or\n<\/p>\n<pre>requestedOrientation = ActivityInfo.SCREEN_ORIENTATION_PORTRAIT \/\/ Kotlin<\/pre>\n<p>\n  For the orientation-locked app to remain in portrait mode, Surface Duo keeps the app\u2019s orientation and creates a \u201cletterbox effect\u201d to preserve the correct aspect ratio.\n<\/p>\n<p>\n  This is a new situation for many apps, and because the app is now oriented differently to the device, the camera view will be rotated. As a developer, you may want to tweak your code for a better dual-screen experience. There are a few options to consider:\n<\/p>\n<ol>\n<li>\n  Detect the letterboxed state, show a message to the user asking them to rotate or flip the device for a better experience.\n<\/li>\n<li>\n  Detect the letterboxed state and rotate the camera feed to match the device posture.\n<\/li>\n<li>\n  Remove the portrait orientation lock. This would allow the app to rotate freely, but if your UX depends on the portrait orientation, it might not be an option for you. \n<\/li>\n<\/ol>\n<p>\n  The sample app includes code for each of these options, explained in more detail below.\n<\/p>\n<h3>1. Show a message. <\/h3>\n<p>\n  To recognize when an orientation-locked app has been \u201cletterboxed\u201d, we have created a helper class <a href=\"https:\/\/github.com\/microsoft\/surface-duo-sdk-samples\/blob\/master\/Camera\/src\/main\/java\/com\/microsoft\/device\/display\/samples\/camera\/PortraitLockHelper.java\"><strong>PortraitLockHelper<\/strong><\/a>. This is configured in the activity\u2019s <strong>OnCreate<\/strong> method by wiring up a listener and checking for a letterboxed state being detected:\n<\/p>\n<pre>portraitHelper = new PortraitLockHelper(this);\r\nportraitHelper.StateListener = new PortraitLockHelper.PortraitStateListener() {\r\n    @Override\r\n    public void PortraitStateChanged(int state) {\r\n\/\/...\r\n    if((state & PortraitLockHelper.PORTRAIT_STATE_LETTERBOXED_90) > 0 ){\r\n        if(showRotationMessage){\r\n            rotationMessageView.setVisibility(View.VISIBLE);\r\n        }\r\n\/\/...\r\n    }\r\n}<\/pre>\n<p>\n  In the <a href=\"https:\/\/github.com\/microsoft\/surface-duo-sdk-samples\/tree\/master\/Camera\">Camera sample<\/a> we use this information to show a message advising the user to rotate the device. This is the simplest solution that preserves the app\u2019s portrait-locked behavior without requiring any other significant code changes.\n<\/p>\n<h3>2. Rotate the camera feed<\/h3>\n<p>\n  If you\u2019re interested in better supporting the \u201cletterboxed\u201d state by correcting the orientation of the camera feed, you\u2019d use the same <strong>PortraitLockHelper<\/strong> class to recognize that the app has been \u201cletterboxed\u201d. However, instead of showing a message to the user, rotate the camera data stream so it matches the app\u2019s perceived orientation.\n<\/p>\n<p>\n  For this to happen, you need to make sure you use a <a href=\"https:\/\/developer.android.com\/reference\/android\/view\/TextureView\">TextureView<\/a> and not a <a href=\"https:\/\/developer.android.com\/reference\/android\/view\/SurfaceView\">SurfaceView<\/a> as your preview area. \n<\/p>\n<p><a href=\"https:\/\/developer.android.com\/reference\/android\/view\/TextureView\">TextureView<\/a> allows the developer to transform it, thus enabling the ability to rotate and stretch the content. For example, textureView.setRotation(90) will rotate the feed 90 degrees. And to stretch the feed (to compensate for the change is aspect ratio), the setScaleX and setScaleY methods are used.\n<\/p>\n<p>\n  In the sample we use the following values:\n<\/p>\n<table>\n<tr>\n<td>\n<p><strong>State<\/strong>\n<\/p>\n<\/td>\n<td>\n<p><strong>Feed Rotation<\/strong>\n<\/p>\n<\/td>\n<td>\n<p><strong>Scale X<\/strong>\n<\/p>\n<\/td>\n<td>\n<p><strong>Scale Y<\/strong>\n<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>\n  default\n<\/p>\n<\/td>\n<td>\n<p>\n  0\n<\/p>\n<\/td>\n<td>\n<p>\n  1\n<\/p>\n<\/td>\n<td>\n<p>\n  1\n<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>\n  Flipped\n<\/p>\n<\/td>\n<td>\n<p>\n  0\n<\/p>\n<\/td>\n<td>\n<p>\n  1\n<\/p>\n<\/td>\n<td>\n<p>\n  1\n<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>\n  Spanned\n<\/p>\n<\/td>\n<td>\n<p>\n  90\n<\/p>\n<\/td>\n<td>\n<p>\n  4\/3\n<\/p>\n<\/td>\n<td>\n<p>\n  4\/3\n<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>\n  Letterboxed 90 deg\n<\/p>\n<\/td>\n<td>\n<p>\n  90\n<\/p>\n<\/td>\n<td>\n<p>\n  4\/3\n<\/p>\n<\/td>\n<td>\n<p>\n  4\/3\n<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>\n  Letterboxed 270 deg\n<\/p>\n<\/td>\n<td>\n<p>\n  270\n<\/p>\n<\/td>\n<td>\n<p>\n  4\/3\n<\/p>\n<\/td>\n<td>\n<p>\n  4\/3\n<\/p>\n<\/td>\n<\/tr>\n<\/table>\n<p>\n  The sample app also includes buttons to enable or disable the camera feed rotation, which helps demonstrate the rendering difference.\n<\/p>\n<p>\n  The code to change the rotation setting is in the <strong>PortraitStateListener<\/strong> declaration. First it determines the device rotation, and whether the camera feed should be rotated. It then calls the <strong>transformText<\/strong> method which calculates and applies the scale and rotation values to suit the device and application settings.\n<\/p>\n<h2>3. Support all orientations with Camera2 <\/h2>\n<p>\n  If your app supports both portrait and landscape orientation, the Camera2 API can adapt to both. Rather than lock the orientation, your app should allow the view to adapt to both portrait and landscape and adapt the feed accordingly.\n<\/p>\n<p>\n  The <a href=\"https:\/\/github.com\/microsoft\/surface-duo-sdk-samples\/tree\/master\/Camera\">CameraSample<\/a> contains the code to support orientation changes with a camera view. The code path is similar to rotating the camera feed, except that first it will change the requested orientation (e.g. to allow any orientation):\n<\/p>\n<p><strong>Java<\/strong>\n<\/p>\n<pre>\r\nsetRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_UNSPECIFIED);\r\n<\/pre>\n<p>\n  It then calls <strong>PortraitStateChanged<\/strong> which does the same determination of device state before applying scale and rotation values. Apps that choose not to lock to specific orientations can use this code to create a camera view that works in all orientations.\n<\/p>\n<h2>CameraX<\/h2>\n<p>\n  If you\u2019re building a new app or planning to upgrade your camera code, adopting the <a href=\"https:\/\/developer.android.com\/training\/camerax\">CameraX API<\/a> will give you a modern base to build on. Google\u2019s <a href=\"https:\/\/github.com\/android\/camera-samples\/tree\/master\/CameraXBasic\">CameraXBasic sample on GitHub<\/a> runs great on the Surface Duo \u2013 but we have two small code tweaks to improve the experience:\n<\/p>\n<p><strong>AndroidManifest.xml<\/strong>\n<\/p>\n<pre>\r\nandroid:configChanges=\"orientation|screenLayout|screenSize|smallestScreenSize\"\r\n<\/pre>\n<p><strong>MainActivity.kt<\/strong><\/p>\n<pre>    override fun onWindowFocusChanged(hasFocus: Boolean) {\r\n        super.onWindowFocusChanged(hasFocus)\r\n        if(hasFocus){\r\n            container.postDelayed({\r\n                container.systemUiVisibility = FLAGS_FULLSCREEN\r\n            }, IMMERSIVE_FLAG_TIMEOUT)\r\n        }\r\n    }<\/pre>\n<p>\n  These minor updates provide a smoother experience when spanning and unspanning the app, and when focus changes between the camera app and whatever is on the other screen.\n<\/p>\n<h2>Feedback<\/h2>\n<p>\n  For existing apps with camera functionality, this post has two different options for how you might enhance the user interface for the Surface Duo, with working sample code. If you\u2019re building a new app, you can base it on the sample using the Camera2 API, or use the Google sampled based on CameraX.\n<\/p>\n<p>\n  We would love to hear from you about your experiences using the camera in your Surface Duo apps. \n<\/p>\n<p>\n  Please reach out using the <a href=\"http:\/\/aka.ms\/SurfaceDuoSDK-Feedback\">feedback forum<\/a> or direct message me on <a href=\"https:\/\/twitter.com\/tggmbi\">Twitter<\/a>. <\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hi Android developers! While the Surface Duo runs Android apps just like any other Android device, there are some differences in the hardware: There is one camera, which can behave as either front-facing or rear-facing depending on how the device is folded. Tapping the \u201cswap cameras\u201d button will change how the image is mirrored to [&hellip;]<\/p>\n","protected":false},"author":23662,"featured_media":1014,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[703,45],"class_list":["post-1013","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-surface-duo-sdk","tag-camera","tag-surface-duo-sdk"],"acf":[],"blog_post_summary":"<p>Hi Android developers! While the Surface Duo runs Android apps just like any other Android device, there are some differences in the hardware: There is one camera, which can behave as either front-facing or rear-facing depending on how the device is folded. Tapping the \u201cswap cameras\u201d button will change how the image is mirrored to [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/posts\/1013","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/users\/23662"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/comments?post=1013"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/posts\/1013\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/media\/1014"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/media?parent=1013"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/categories?post=1013"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/tags?post=1013"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}