Beginning last fall, I had the opportunity to work with the Surface Duo OneDrive feature team to redesign and polish the OneDrive app user experience to make it shine on Surface Duo. Many thanks and kudos to the team:

Eli Waalkes, Daniela Dimitrova, Hansel Ip, Eirini Tsantouli, James Tong, Tola Adeola, and Lin Wang.

Now that Surface Duo has been released, we are excited to share the changes we have made and lessons learned on this journey. Hopefully, these tips will helpful to the developer community.

The smart phone industry has evolved rapidly. Every year, there are new phone OS upgrades released, and plenty of new hardware designs have emerged on the market. Along with OS and hardware advancements, tons of apps have evolved at the same time to support these new hardware form factors. In addition to app feature development, app developers need to adapt their apps to new OS versions and support a broad spectrum of hardware. When it comes to screen display, throughout the mobile phone history, it was assumed that a user always uses a single screen, whether large or small. Although device hardware may vary, as long as the same OS is used, apps can be easily ported to other devices without requiring major design changes.

However, this was not the case when Microsoft introduced the Surface Duo device to the market, which provides a unique user experience with its dual-screen form factor. Without any change, an app would work well on a Surface Duo device when the app occupies only a single screen. In order to fully empower users with the capabilities of Surface Duo’s beautiful dual-screen display, app developers will need to make design improvement to provide the best user experience.

In this blog post, we will showcase the Microsoft OneDrive app, and illustrate what design changes were made to empower users to leverage the dual-screens on Surface Duo as well as some challenges we faced.

Dual-screen phone productivity

Surface Duo has a foldable design with two screens. Each screen on Surface Duo is very wide, and feels like a mini tablet. The wide space allows a user to comfortably view images, videos, email, web site content, etc. This beautiful screen is duplicated on Surface Duo, providing the user two large viewing screens on a single device. This enriched viewing experience is like a PC with dual monitors, but in a users’ hands. A user can play video on one screen while doing some other task on the other screen. As Surface Duo is more productive than a single screen device, I find myself spending a lot more time using Surface Duo than any other mobile device I have used in the past. With productivity in mind, most of the design improvements we made in the OneDrive app for Surface Duo was around leveraging the dual-screen form factor.

Single screen experience

Consistent Experience with any Single Screen Device

There are two scenarios for an app to occupy one screen:

• Single screen device scenario

  
  Figure 1: Surface Duo in single-screen mode

  Surface Duo has a unique hinge design such that a user can open the phone 360 degrees and use the device as a normal one-screen device, with only one screen turned on and the other screen turned off.

• Dual-screen device scenario

  When the device is unfolded, two screens are presented to the user. When an app is launched on Surface Duo, it always occupies just one screen.

For single screen scenarios on Surface Duo, we preserved the same UI experience as when running an app on any other Android device with no design changes.

Enhanced app to app interaction on dual-screen devices

On Surface Duo, when the device is unfolded, a user can open OneDrive on one screen and launch another app on the other screen. This opens up the opportunity for the user to to copy data from one screen to the other, or vice versa.

To enhance the user experience for this scenario, we added drag and drop functionality in the OneDrive app to allow users to easily copy data out of OneDrive or into OneDrive from an app being used on the opposite screen. For instance, a user can select and drag an image file from his or her OneDrive app, then drop it to the other screen where the user is composing an email so that the image can be attached to the email message.

The opposite direction of data flow is also supported in the OneDrive app, enabling a user to pull files or photos from other apps into the OneDrive app, provided that drag-and-drop is also supported in the other app.

Figure 2: Surface Duo app usage in dual-screen mode

Dual-screen experience

Traditional user experience

As a cloud storage app, OneDrive has a rich feature set that allows users to manage their storage and access their files in their OneDrive account from anywhere. The most common user scenarios are:

• Storage browsing – The user can browse their file structures, most recent accessed files, shared files, business document libraries, photos, and media content.
• File content viewing – The user can view their documents, photos and media files with OneDrive app.

Traditionally, the OneDrive mobile app’s design only allows one task at a time. Different tasks are mutually exclusive. For instance, you can browse the contents of a folder, then you can open a file to view the contents. What if you want to browse other files in the same folder while you are viewing content from a different file? For that scenario, the user would have to exit the current viewing experience first.

Enhanced files experience

On a Surface Duo device, a user can span the OneDrive app from a single screen onto two screens.

We introduced a list-detail layout for this scenario, so that we can enable both storage browsing and file content viewing experiences at the same time. For instance, the user can browse a folder content on the left screen, and then tap on the right screen to view the file. The user can also perform additional operations like browsing or sharing on the left screen without exiting the file content view on the right screen.

Figure 3: Surface Duo OneDrive app spanned across dual-screens enabling user multi-tasking

Immersive reading

For users who want to focus on reading files, they can enter an immersive reading mode by tapping the file spanning button so that the file will be shown on both screens: one page per screen, like a book. The user can flip through pages easily by swiping the screen to advance to the next page in the file. With two hands holding the foldable device, it feels like reading a physical book.

Figure 4: Surface Duo immersive reading experience across dual-screens

Enhanced photo experience

As camera roll backup becomes popular among OneDrive feature set, the OneDrive photo viewing experience has been very crucial to customers. Similar to a files browsing experience, by leveraging the list-detail layout, a user can browse their photos on the left screen, and easily select and view a photo on the right screen. The user can also zoom a photo or play a media file. The user can swipe the collection on the right screen in order to view its adjacent photos or media content within the same viewing collection.

Figure 5: Surface Duo photo browsing experience across dual-screens

While the user swaps photos on the right screen, we maintain the browsing view on the left screen, automatically scrolling through the collection so that the photo or media file that is currently being viewed on the right screen is shown on the left, so the user won’t lose context.

List-detail design deep dive

Now let’s dive into some details about our design to optimize OneDrive for dual-screen experiences.

Enable content browsing and file viewing experiences at the same time

List-detail layout

In order to enable a OneDrive browsing experience while viewing the contents of a selected file experience, it is a natural choice to adopt list-detail layout so that the folder structure can be presented on the left screen, and the user selected file can be shown on the right screen. The code snippet shown below illustrates the list-detail layout structure, which includes three sub view containers:

• List view container

  This is a container (with android:id value as “@+id/main_container_master”) to host OneDrive storage browsing experiences such as the user’s OneDrive folder structures, the users recently accessed files, shared files, photos and media views, offline content, etc.

• Detail view container

  This is a container (with android:id value as “@+id/main_container_detail”) to host OneDrive file viewing experiences such as document rendering, photo view, media file player, as well as any file type that previewing is supported.

• Hinge view

  This view (with android:id value as “@+id/main_container_divider”) represents the Surface Duo device hinge space between the two screens. Any UI elements that fall into this area won’t be visible to the user. The Surface Duo SDK provides an API to allow developers to identify such areas and handle it accordingly based on app scenarios. We included this view block in the list-detail layout to ensure neither browsing UI nor file previewing UI fall into the space of the Surface Duo device hinge.

The layout dimension is specified when the hosting activity class is created. As illustrated in the code snippet below, we ensure that the hinge view (with android:id value as “@+id/main_container_divider”) occupies the exact position and space of the hinge area, so that the list view container occupies the left screen and the detail view container occupies the right screen precisely.

Transition to fragment-based implementation

The list-detail layout design requires that UI components to be loaded into list view container or detail view container need to be implemented as Fragment subclasses.

Prior to Surface Duo feature work, many OneDrive UI experiences such as storage browsing and file viewing used to be implemented as Activity subclasses. Navigation between different UI experiences are implemented in a way to transit from one Activity instance to another. For instance, when a user browses a folder content, they tap a file for viewing. This will trigger a transition from folder browsing Activity instance into a file viewing Activity instance.

For the new experience on a Surface Duo, with the list-detail layout, both the folder browsing and the file viewing experiences are shown within a single Activity (MainActivity class) instance and navigation events will trigger fragment adding or replacing transactions. We did some conversion work to transform Activity-based implementation into Fragment-based implementation so that they can be loaded into the list-detail layout when running on Surface Duo devices.

Converting from Activity-based implementation into Fragment-based implementation involves work in many areas. At a high level, here is a list of work items developers need to take care of to optimize an app for use on a dual-screen device:

• UI conversion

  Transition from Activity UI to Fragment UI should be straight forward. However, when it comes to the Surface Duo device, you may need to consider some additional factors. For instance, if your app has Activity level action toolbar on the UI, you may want to move it to fragment level toolbar implementation and limit it to be shown only on one screen when your Activity spans to two screens.

  If your app applies dynamic layout based on screen width DP values, you may need to add some handling to ensure that your Fragment level layout is based on one screen width instead of two-screen widths when your app is spanned across two screens.

• Behavior handling and fragment lifecycle handling

  Any data loading, UI rendering, user interaction event handling, and any additional behaviors need to be moved from your Activity level classes into Fragment level classes. Similar to Activity lifecycle event handling, Fragment lifecycle events need to be handled properly in the Fragment level classes.

• Navigation handling

  With list-detail layout, some navigations, like tap to open a file, will not trigger to start a new activity. Such navigations will happen within the same Activity instance and hence it needs to be implemented as loading or replacing a fragment within the detail container.

  Similarly, backward navigation also needs to happen at the Fragment level.

• Fine tune the experience between two screens

  Although the left screen and right screen show different things to the user, a proper link between the two may provide the user a better experience. For instance, in OneDrive photos view, we allow the user to swap the screen to view adjacent photos or media files. We added a notification mechanism so that the browsing fragment on the left screen can monitor what file the user is currently viewing on the right screen and can scroll to the file nearby position to help the user to keep the viewing context easily.

Managing screen configuration changes

Screen configuration states

When running apps on a single screen device, we manage only a single factor of screen configuration change: screen orientation, allowing a user to rotate the device. Whether its portrait orientation or landscape orientation, we have proper handling for that UI experience.

Figure 6: Surface Duo screen orientation states

Surface Duo brings in a new configuration factor: app spanning state, which indicates whether the app occupies one screen or both screens. The user can use a finger to drag and hold the app from the bottom and glide to near center of the device so that the app will be shown as spanned across two screens. The user can perform an opposite operation to return the app to a single screen.

The spanning and unspanning operations can also happen when the device is in portrait orientation. With permutation of the two factors, we have four screen configuration states as illustrated in the diagram.

Layout choices and list-detail visibility

Apps need to handle these screen configuration changes properly in order to render the UI as desired for whatever screen orientation is chosen. For some configuration states, we want to show both a browsing view experience and file viewing experience; for other configuration states, we only want to show one of the experiences depending on the situation. When we first started working on design changes to OneNote to optimize the app for dual-screens, we evaluated a scheme to apply a different layout for different configuration states when the Activity instance is created:

• Apply list-detail layout when we intend to show both a browsing experience and a file viewing experience, i.e. when the app is spanned and the device is in landscape orientation.
• Apply single screen layout in other cases.

Then, we quickly figured that this scheme may lead to a very complex and messy logic when it comes to handling various state transition paths, together with ensuring a consistent user experience for backward navigation behavior.

Our current design is to always use list-detail layout for all four screen configuration states and manipulate the visibility of the list-detail container based on the app screen configuration state.

• Show both list-detail view containers (all-view-visible) when the app is spanned and the device is in landscape orientation (per our current design).
• For the other screen configuration states, we show one sub-view (single-view-visible), either the list-view container or the detail view container and hide the rest.

Figure 7: Surface Duo app experience in spanned and un-spanned modes

When we transition from all-view-visible state to a single-view-visible state, we need to be smart about which sub-view shows and which sub-view to hide. For instance, when a user unspans the OneDrive app from two-screens into one screen when the device is in landscape orientation, we need to determine whether to show a storage browsing view or a file content view. In the diagram above, because the user did not select any file to view, after the unspanning transition, we just show the folder browsing experience in the list-view container, and hide the hinge-view and the detail-view container.

In the example below, because the user was viewing a selected photo while the app was spanned across two screens, after the unspanning transition, we show the photo in the detail-view container, but hide the hinge-view and the list-view containers.

Figure 8: Surface Duo OneDrive browsing experience in span and unspanned modes

Handle backward navigation

Navigation on single screen vs dual-screen

Android devices’ back key design allows a user to easily go back to the last viewing experience the user had visited. Typically, the existing Android SDK framework will be help you handle this naturally with ease, either through Activity navigation or through the FragmentManager’s backstack mechanism.

With Surface Duo, the transition path can be a bit more complex. For instance, the user may perform navigation on one screen, then span the app to two screens, then navigate around, then rotate the device or unspan the app to one screen. The backward navigation may get a bit weird if not designed properly. Here we illustrate some of the handling we did in the OneDrive app.

Assuming your OneDrive has two folders: Folder-1 and Folder-2, and Folder-1 has three files. The user navigates to Folder-1 and selects a file (myPdf-11.pdf) to show. The experience on a single screen device will be like this:

Figure 9: Surface Duo navigation experience in OneNote in single-screen mode

When using OneDrive on a Surface Duo, when both browsing view and file preview are shown together, with the same sequence of actions, the experience of navigation forward and backward will look quite similar.
Figure 10: Surface Duo navigation experience in OneNote in dual-screen mode

However, because both folder browsing view and the file content view are visible at the same time on a Surface Duo device, the user can open a different file with a single tap, whereas on a single screen device, the user will have to perform two operations: exit the file content view and then open another file.

Then after the user opens the second file, what should happen if the user taps the back button on a Surface Duo device? Should it go back to the last file the user just viewed?

We can make it back to last file the user just viewed, but the behavior looks a bit out of context if the user first unspanned the app to one screen before tapping the back button, because backing to a different file viewing UI cannot happen if all the operations happen on a single screen, and the user would expect to just go back to the folder browsing view. We can decide to let it go back to the view that shows the folder content and clear the detail view, so that users will have a consistent user experience across all screen configuration states with various state transition paths.

On the implementation, we leveraged FragmentManager’s backstack mechanism within the Activity instance, and backward behavior can happen naturally for most scenarios. To handle backward navigation properly for a file content viewing experience as described above, we did our own customized handling:

• When adding the file viewing fragment into the detail view container, do not invoke .addToBackStack() on the fragment transaction because we don’t want to add the file viewing fragment into backstack.
• Intercept the back button tap event and add customized logic. When a back button tap event is detected, if the detail-view container is not empty, we clear the detail-view container; otherwise, we leverage the existing back button pressed event handling logic so that the rest of the scenarios can happen as expected.

Resources and feedback

We would love to hear from you about your experiences using the Surface Duo SDK, emulator, and your thoughts on how you can utilize these in your apps.  

Please reach out using our feedback forum or on GitHub.