Experimenting with postures in Flutter

Andrei Diaconu

The things I’m doing in this article are highly experimental. By the end of it hopefully you will agree with these two statements:

1. Foldable devices are playgrounds for your new interaction ideas.
2. Flutter is great for prototyping your new interactions.

To achieve the desired behavior, I end up using several hacky techniques which I would think twice before shipping to production. This article is not a tutorial. It is a story of the fun I had with flutter and Surface Duo. Proceed with caution, you have been warned 😊.

What I built

First let’s have a look at what I am trying to achieve. This video walks you through the idea; I also explain the idea in text and image in this section:

Imagine a two-player turn-based game where opponents sit facing each other. The Surface Duo stays in the half-opened posture (like a mini laptop) for the whole duration of the game. It is player one’s turn and both screens are facing the player. Player two can’t see much of what player one is doing since they are facing the back of the device.

When it is their turn, player two flips the device over, dragging the top screen until it becomes the new bottom screen. While in motion, when both players can see the screens, the UI shows a blank white screen, so as not to leak any information to either player.

Once the device has been flipped, it is now showing the contents meant for player two. Player one can’t see anything since they are facing the back of the device. It is now player two’s turn.

What I think makes this idea fun is that players share the device. I remember split-screen games from when I was a child and wanted to bring some of that nostalgia back. Players being in the same room adds to the experience. Using the same device this way makes the experience unique and memorable. The device is part of the game.

First challenge: Who’s turn it is

Let’s start with the challenges we face. First challenge is knowing which player is looking at the screens. Your intuition might tell you that we need to use sensor data to achieve some of this functionality. This is not where it gets complicated.

Initially I thought that the gyroscope would give me the data I needed. It reports the speed at which the device rotates on each axis. My thinking was that I would classify the rotation gesture in one direction as player one and the opposite of that as player two. It did not pan out as I hoped, mainly because this sensor reports speed. If a player takes their time while tipping the device, the gyroscope can record next to no data. It worked some of the times, but it was very unreliable. I needed something else.

I decided to use accelerometer sensor data to figure out if the screen touching the table is the screen with the camera (right) or the screen without the camera (left). Accelerometer data tells the app which way is down, since gravity is constant acceleration downwards. This is how I decide which player looks at the screen, regardless of device orientation. When it is not their turn, player two always sees the camera on the back of the device and player one always sees the simpler screen. I name them “screen with camera” and “screen without camera” because right and left start to lose meaning in the next part.

The way I packaged this for the rest of the app to consume is a good, old, trusty InheritedWidget called HalfOpenedOrientation. Data exposed by it has screenWithCameraPosition, an enum that can be either flatOnTable or uprightVertical. The game I made looks at this value and decides if the current player is one or two. The interesting bit of code, the one that calculates this value based on accelerometer data, looks at how well the x or z axis of the sensor aligns to the gravitational pull. Some trial and error went into making this work on my device, which is why this is of experimental quality. Other devices might have their sensors aligned differently inside the device, so a step further for this would be to do some calibration when the game starts. If you’re close to the calibration data, it’s player one, if you’re far, then it’s player two, for example.

I Initially also introduced a degree of confidence for the screenWithCameraPosition value. If the confidence is low, maybe don’t show anything on the screen, since it seems we are in between positions. After experimenting with a double value for this, I decided it is too much for consumers to know about and reduced it to a screenWithCameraPositionUndecided boolean. When it is true, the game should consider neither player one nor player two has full privacy.

Second challenge: Which way is up

Something less obvious is that this idea challenges the concept of device orientation. By the end of this chapter, you might think “If you say portrait one more time…” but nonetheless, this seems interesting to discuss. My first plan was to block the app in portrait mode using the android manifest config and rely on the OS to make sure the UI is not upside down.

When you span an app blocked in portrait mode, Surface Duo does what it can to oblige. The definition of portrait is that height is bigger than width. The joint width of both screens on Surface Duo exceeds the height of one screen. This means that the canvas covering both screens sits in “portrait” when the hinge is horizontal! That surprises many people, so stop to think about it. We sometimes call this “dual-landscape”, because the individual screens are in landscape and your app occupies both. The resulting canvas is portrait, though, as defined by both Flutter and Android (height is bigger than width).

Consequently, the first step you will take to enhance your Android app for foldables will be to remove the android:screenOrientation="portrait" if you have it in your manifest. Otherwise, your app will turn sideways when it is spanned, not only on foldables, but on all tablets as well.

With the app blocked in portrait, my next step was to see if the device reverses the portrait to upside down portrait the way that I want. If you hold the device in “dual-landscape” and rotate it, your app gets rotated to match. But was this behavior sensitive enough to flip the UI when player two merely tips the device? The answer is no, and we like it this way. The device must strike a balance when making these switches so as not to become finicky, and what my game needs is too sensitive for normal, daily use. But can I rely on it to not change at all, then? The answer is still no. Players can handle the device in all kinds of ways, perhaps even shaking it during a heated dispute. I can’t rely on the OS mechanism flipping threshold to either stay the same or change on each device tip.

This is where it gets complicated. Let’s assume that the UI looks good for player one and then player two tips the device their way, but the OS does not detect the orientation flip. Player two now has exclusive access to the screens, but everything is upside down. The first solution for this was to force the Android app to request a certain orientation when my accelerometer logic kicked in. I decided that I may want this to work as a Flutter Web app at some point, so maybe simply rotating the UI using Transform wasn’t such a bad alternative.

The result was BuoyantTwoPane, a widget similar to TwoPane that takes topPane and bottomPane parameters, and no matter how you rotate the device, it makes sure that topPane stays at the top in the real world. It’s a crazy widget, I admit, but it does the job. While playing the game, you can easily end up with the OS upside-down (the status bar at the bottom) but the game still faces the player the correct way. You can see this in the video at the beginning if you look specifically for the status bar position. The name suggests that the layout is buoyant, much like a fishing float which always stands upright in water. This article and project fall under the experiments umbrella, so don’t think this is in any way normal behavior that you should expect in one of our next OS updates.

Putting things together

Now that we know who’s turn it is and we don’t have to worry about orientation, the rest of the steps are simple. I didn’t flesh out the rest of the game and simply had my proof of concept show a big “1” or “2” on the screen, and also give some instructions when the device is not in the correct posture. To know posture is, my app looks at MediaQuery. If you want to learn more about what support Flutter offers for foldable devices, we have documentation on MediaQuery that might interest you. That and TwoPane are our normal offerings. They don’t float or rotate in crazy ways, you can rely on them when you enhance your app for all foldable devices (not just Surface Duo).

Call to action

1. Look at the code and fork it if you want to build the rest of the game.
2. Learn more about foldable support in Flutter by looking at MediaQuery.
3. If you have any questions or would like to tell us about your dual-screen applications, use the feedback forum or message us on Twitter @surfaceduodev.
4. Finally, please join us every Friday on Twitch at 11am Pacific time to chat about Surface Duo developer topics!

Andrei Diaconu Senior Software Engineer, Surface Duo Developer Experience

Follow