My inspiration

Back in 2018, I came up with a way of using a Microsoft Surface to create 360 panoramic sketches and the Microsoft Garage encouraged me to release it as an app, Sketch 360. I created Sketch 360 because I wanted to develop a tool to help me create these immersive sketches more quickly. Because they are sketches from a single point looking out in all directions, they take more time to draw than a typical sketch. Sketches are supposed to be quick and I wanted to make the process faster for 360 sketches.

Since its initial release, the free app has been downloaded more than 41,000 times from the Microsoft Store by Windows users. I’ve made a number of updates to the app, adding features like 360 video exporting, stencils to help you draw along the curved equirectangular grid, layers, inserting images and videos, and an undo/redo button. I’ve given several talks and have run workshops about the app and the technique of using an equirectangular grid to users construct an accurate perspective drawing from a single point looking out in all directions. These drawings can be easily shared on the web on sites like Facebook, Kuula.co, Adobe® Photoshop® Lightroom®, and YouTube as 360 photos and videos because the app adds the requisite metadata so that the images or videos are viewed as 360 photos or videos. If you import a 360 sketch into Adobe Photoshop software as a Spherical Panorama, you can edit it “in the round” with the full functionality of Adobe Photoshop software.

Figure 1: Sketch 360 on Windows

Figure 2: 360 Sketch turned into a 360 video

I chose to develop Sketch 360 for Windows and Microsoft Surface because of the great support for inking as an input device and my familiarity with the Windows UWP platform, having built numerous creative apps for it. Though I saw that the Sketch 360 had appeal for VR designers, game designers, architects, storyboard artists, and urban sketchers like me, one thing that I noticed that many of these creative people didn’t have Windows devices with pens like Microsoft Surface devices.

Figure 3: Sketches from my travel journal

Surface Duo: a dual-screen creative tool with pen support

I created Sketch 360 to help me, a person educated as an architect who has spent thirty years keeping journals, create 360 degree drawings. I built the Windows version of Sketch 360 for an audience of one but found that many others would find it useful. The Microsoft Garage enabled me to release it as an experiment and from that, I realized that I needed to build the app for more platforms. With the upcoming release of Surface Duo, I saw a unique opportunity for Sketch 360: not only did it support the pressure-sensitive Surface Pen, but it also had a dual-screen interface which matched very well with the two-pane UI of Sketch 360 on Windows.

If I rebuilt Sketch 360 using Xamarin.Forms, I could target Surface Duo, other pen-native Android devices like the Samsung Galaxy Note, and any other modern Android devices as well as Apple iOS devices with a single app; perfect for a “passion-project” like Sketch 360 with very limited time and resources required.

I love creative tools – my favorites are actually analog: pens, pencils, watercolors, and paper. My aim with digital tools is not to replicate the analog, but go beyond it; my favorite app right now, Adobe Fresco®, does that beautifully, turning my Surface Pen into a watercolor brush, a soft pastel, or a rough pencil. I wanted to enable a type of creativity that is difficult or impossible with traditional pens, pencils, or brushes, like a 360 sketch. My inspiration for creating digital tools is often rooted in my analog creative skills and techniques. Most importantly, digital tools can enable creativity in locations that are impossible or impractical to bring analog tools, like the cockpit of an airplane or in a spacecraft.

Figure 4: my travel journaling supplies

Throughout my career, my passion has been for building and using creative tools so my imagination comes alive with a dual-screen, pen/touch capable device that folds out from my pocket to the size of my typical journal page. I was definitely up for the challenge of moving my native C# UWP app to a C# Xamarin Forms. When Microsoft Garage sponsored an internal hackathon to encourage employees to build or adapt apps for Surface Duo, I jumped at the chance. Having two separate input surfaces on Surface Duo opened the creative possibilities: each screen represented the same thing, but from a different perspective. My favorite “posture” for using Sketch 360 on Surface Duo is having the drawing surface flat on the table and the 360 view tilted up, but I could easily adapt to a standing posture with the drawing surface on the left and the 360 view on the right (since I’m left-handed).

Figure 5: Sketch 360 on a Surface Duo

Xamarin Forms + SkiaSharp

The architecture of the app consists of two sides, a drawing canvas, and a 360 view. Every time the user adds a stroke, the 360 view is updated and pans to the ending point of the stroke just added. The drawing canvas on Windows used the native InkCanvas control which made it easy to collect and render ink. Having created Sketch 360 once in C#, I knew just how to build it again in C# with a smaller feature set, optimized for mobile devices using Xamarin Forms. In moving to Xamarin Forms, I needed to rewrite most of the C# and Xaml code and needed to find something to provide ink input since there was no analog to the InkCanvas for Xamarin Forms. For that, I created a InkCanvasView using SkiaSharp which handled inking input and rendering. I tried my best to make it like the UWP InkCanvas in terms of API surface, implementing just enough of it to support the Sketch 360 core scenarios:

• Drawing and erasing ink strokes
• Pressure-sensitive ink strokes with multiple colors and variable thickness
• A virtual canvas that supports panning and zooming
• Wet stroke updating to enable curved guides while drawing
• Loading and saving vector ink data

The Windows Sketch 360 version uses the Windows Ink Serialized Format for ink data serialization. Though the format specification is available to be used in non-Windows apps, I did not have the time to write a parser for my app. Instead, in Xamarin Forms I serialized ink to a simple JSON format and modified the Windows version to be able to read that format in addition to the Windows ink. This means that you can start a sketch on your phone, save the .sketch360 file to your OneDrive, and open it up again on a larger Surface device to complete it.

Figure 6: Sketch 360 User Interface with the 360 view on top and drawing surface on the bottom

Xamarin Forms development

The new Xamarin Forms TwoPaneView API for dual-screen devices made it easy to define a UI that adapts fluidly from a single pane to two, portrait, or landscape. Here is a simplified version of the Xaml for the two pane UI supporting both landscape and portrait for both single-pane and spanned:

<dualscreen:TwoPaneView x:Name="TwoPaneView" PanePriority="Pane1">
    <dualscreen:TwoPaneView.Pane1>
        <Grid>
            <views:DrawingView/>
        </Grid>
    </dualscreen:TwoPaneView.Pane1>

    <dualscreen:TwoPaneView.Pane2>
        <Grid>
            <views:SphericalView/>
        </Grid>
    </dualscreen:TwoPaneView.Pane2>
</dualscreen:TwoPaneView>

Xamarin forms made the process of creating a cross-platform mobile creative app straightforward, and with the additional tools in the Xamarin Essentials, I was able to make it as functional and performant as a native app. For most of the development, using the Surface Duo Emulator enabled my volunteer team of hackers and me to build the app without having a physical device in-hand.

Babylon.JS

The 360 view is a WebView control using Babylon.JS, an open-source JavaScript 3D framework which has a PhotoDome API. The PhotoDome makes it easy to render an equirectangular image on the inside of a sphere, providing the touch, gyroscopic-sensor enablement and VR-capabilities for 360 viewing. I was able to use the HTML/JavaScript code using Babylon.JS with very little modifications from the Windows version – which made the job much easier. If you want to try the Babylon.JS PhotoDome code in a browser with your own image or sketch, take a look at this site that I created to use in the workshops that I have run to teach people about 360 sketching. Just load it with your own 2×1 aspect ratio image, no matter which tool you created it with.

Learning 360 Sketching in December

Once you understand how to work with the equirectangular projection, you can start creating some fun sketches with Sketch 360 with your Surface Duo if you pick up a Surface Pen. To help people learn, I have been doing workshops with CODAME an art/tech organization, to teach people both analog and digital 360 sketching. We will be hosting our next Sketching Worlds in 360 online workshop on December 18. Register today, as space is limited.

Available today in the Google Play marketplace

Sketch 360 is available today in Google Play to download and start creating. It works with any modern Android device, but works best with devices that support pressure-sensitive pens like the Surface Duo and Samsung Galaxy Note. Please share any sketches you create with it using the tags @MSFTGarage and #Sketch360.

The source code is also available on GitHub!

Michael S. Scherotter
Artwork | Twitter | Instagram | GitHub | #DearMoon