Hello Android developers!

The Microsoft Surface Duo is a great device for consuming multimedia content, and ExoPlayer is a simple, powerful widget for rendering video. Raymond and I are interns on the Surface Duo Developer Experience team, and we’re excited to share two dual-screen enhanced samples that showcase video experiences on foldable devices.

Our samples were inspired by this Google I/O video Foldables – WindowManager with MotionLayout and the ConstraintLayout FoldableExperiments sample. The goal for each sample is to demonstrate a video player user experience that works great on single-screens, the dual-screen Surface Duo, and other foldable devices.

Figure 1: Adapting to device posture on a foldable emulator

Dual-screen video player

On single-screen devices, video controls are normally placed on top of the video. On foldable and dual-screen devices this could introduce some issues. Hinges can cut off large parts of the video and controls in certain configurations, and hinges can undesirably distort the video. Therefore, it makes sense to sometimes separate the video and the controls, so they are displayed in separate areas on the device.

On devices like Surface Duo, there is a permanent hinge that blocks out a portion of the video. This isn’t too disruptive when the app is spanned in dual-portrait mode, but we can still separate the controls to make them accessible. When the app is spanned in dual-landscape mode, we always want to place the video on the top screen and the controls on the bottom screen to avoid having the video blocked by the hinge. On foldable single-screen devices, there is no “hinge,” but we still may want to separate the controls when the device is partially folded.

Technical details

MotionLayout is used to animate the movement of the PlayerView and external ControlViews. The PlayerView and the ControlViews are constrained to a ReactiveGuide, which changes depending on the presence and location of a hinge/fold.

The app uses Jetpack Window Manager to detect the presence of a fold/hinge when the device’s layout configuration changes. When there is a hinge and the controls need to be separated from the video, the app calculates the position of the fold using the FoldingFeature provided by Jetpack WM’s WindowLayoutInfo and the MotionLayout view. The app pushes that fold position to the ReactiveGuide. The PlayerView and the ControlView are both constrained to the ReactiveGuide position, so they move accordingly to separate the video and the controls. The overlaid controls on the PlayerView are disabled, leaving only the video on one half, and the external controls on the other half. On Surface Duo, this is implemented twice – once for dual-portrait, and once for dual-landscape.

Figure 2: User has control over the viewing experience on Surface Duo – full screen to maximize viewing area or side-by-side with the controls.

Figure 3: Surface Duo “laptop” viewing posture with video on the top screen and controls on the bottom.

inner class StateContainer : Consumer {

  override fun accept(newLayoutInfo: WindowLayoutInfo) {

    fab.hide()

    // Add views that represent display features
    for (displayFeature in newLayoutInfo.displayFeatures) {
      val foldFeature = displayFeature as? FoldingFeature
      if (foldFeature != null) {

        lastFoldingFeature = foldFeature

        //isSeparating is true if there is a hinge, or a fold is apparent in the device
        if (foldFeature.isSeparating) {

          if (foldFeature.orientation == FoldingFeature.ORIENTATION_HORIZONTAL) {
            var fold = horizontalFoldPosition(motionLayout, foldFeature)
            ConstraintLayout.getSharedValues().fireNewValue(R.id.horiz_fold, fold)
            playerView.useController = false

          } else {
            //move over on-video controls so it's not cut off by hinge
            setMargins(connectedControls, connectedControls.width / 2, 0, 0, 0)

            //make split control fab visible
            fab.show()
            //update vertical reactive guide with verticalFoldPosition()
            updateSplitControl(lastFoldingFeature)
          }

        } else {
          //reset fold values
          ConstraintLayout.getSharedValues().fireNewValue(R.id.horiz_fold, 0)
          ConstraintLayout.getSharedValues().fireNewValue(R.id.vert_fold, 0)
          playerView.useController = true // use on-video controls

        }
      }
    }
  }
}

Video player with chat

Video watching experiences often include user participation. This sample focuses on a streaming experience where users often simultaneously watch the livestream and interact with a live chat. The dividing of screen space between two panels, such as a video panel and a chat panel, can work especially well on dual-screen devices.

Layout options

In single-screen situations, the device orientation is enough information to arrange the panels. In landscape mode, the panels should be arranged left-to-right. In portrait mode, the panels should be arranged top-to-bottom. The relative size of the panels is somewhat arbitrary – the animations below show a ratio of 70/30 for landscape and 50/50 for portrait. If the user only wants the video panel, it should fill the screen.

Figure 4: showing and hiding the chat on single-screen

On foldable devices, the orientation of the fold is more important than the orientation of the device. If the fold is vertical, the panels should be arranged left-to-right, and if the fold is horizontal, the panels should be arranged top-to-bottom. The panels should be on opposite sides of the fold and take the width of the fold into account. If the user isn’t using the chat panel, the video panel should only span across the fold if the video experience can be improved (the video gets larger).

Figure 5: showing and hiding the chat on a spanned app

Layout technology

In this sample, the video and chat panels are arranged in a MotionLayout. The layout uses two constraint sets, one for “full-screen” mode, and one for “chat-enabled” mode. Since both top-bottom and left-right arrangements are possible, the chat fragment is duplicated; one lives under the video panel, and another to the right of the video panel. The three panels are constrained to two guidelines, one horizontal and one vertical.

Guidelines can be positioned programmatically, making them perfect for executing the discussed layout ideas. With enough layout information, the program can dynamically control the guideline positions in the “chat-enabled” constraint set. (The “full-screen” constraint set can lock the guidelines to keep the chat panel invisible.)

Like the previous sample, the app uses Jetpack WM to collect layout information each time the device’s layout changes. These updates, along with the user toggling “full-screen” or “chat-enabled” mode, trigger the layout updating functions. MotionLayout automatically animates the transition between constraint sets, giving the app a smooth feel. The code snippet below shows how the root MotionLayout is controlled.

fun setFullscreen() {
    bottomChatView.setPadding(0,0,0,0)
    endChatView.setPadding(0,0,0,0)
    rootView.transitionToState(R.id.fullscreen_constraints, 500)
}

fun setGuides(vertical_position : Int, vertical_padding : Int, horizontal_position: Int, horizontal_padding: Int) {
    bottomChatView.setPadding(0,vertical_padding,0,0)
    endChatView.setPadding(horizontal_padding,0,0,0)

    var constraintSet = rootView.getConstraintSet(R.id.shrunk_constraints)
    constraintSet.setGuidelineEnd(R.id.horizontal_guide, vertical_position)
    constraintSet.setGuidelineEnd(R.id.vertical_guide, horizontal_position)

    if (rootView.currentState == R.id.shrunk_constraints) {
        rootView.updateStateAnimate(R.id.shrunk_constraints, constraintSet, 500)
    }
    else {
        rootView.transitionToState(R.id.shrunk_constraints, 500)
    }
}

Just like the example at the start of the post, the chat can detect and use the fold on single-screen devices as an anchor when showing video and chat together:

Figure 6: Chat viewing experience on a foldable device

Feedback and resources

The code for the FoldingVideo sample and the FoldingVideoPlusChat sample is available on GitHub.

If you have any questions, or would like to tell us about your apps, use the feedback forum or message us on Twitter @surfaceduodev.

Finally, please join us for our dual screen developer livestream at 11am (Pacific time) each Friday – mark it in your calendar and check out the archives on YouTube.