The post Project Austin Part 3 of 6: Ink Smoothing appeared first on C++ Team Blog.

]]>This blog post describes how we perform ink smoothing.

Consider a straightforward ink drawing mechanism: draw straight lines between each stylus input point that is sampled. The devices and drivers we have been using on Windows 8 sample 120 input points per second. This may seem like a lot, but very swift strokes can sometimes cause visible straight edges. Here’s a sample from the app (without ink smoothing) which shows some straight edges:

Here is the same set of ink strokes, but with the ink stroke smoothed.

**Spline**

We are using a spline technique to do real time ink smoothing. Other options were considered, but the spline (a) can be done in real time so the strokes you draw are always smooth as new input points are sampled and (b) are computationally feasible.

There is plenty of literature online about spline smoothing techniques, but in my (limited) research I have either found descriptions that are too simplistic, or descriptions that require a degree in computer graphics to understand. So here’s my shot at something in the middle…

Before computers, a technique was used to create smoothed curves using a tool called a spline. This was a flexible material (heavy rope, a flexible piece of wood, etc) that could bend into shape, but also be fixed at certain locations along its body. For example, you could take a piece of heavy rope, pin the rope to a wall using a bunch of pins in different locations along the rope, then trace the outline of the bendy rope to yield a spline-smoothed curve.

Fast forward several decades and now we are using the same principles to create a smoothed line between a set of points. Say we have a line with many points P0, P1, P2, … To smooth it using a spline, we take the first 4 points (P0, P1, P2, P3) and draw a smooth curve that passes through P1 & P2. Then we move the window of 4 points to (P1, P2, P3, P4) and draw a smooth curve that passes through P2 & P3. Rinse and repeat for the entire curve. The reason it’s a spline technique is that we consider the two points as being ‘pinned’, just like pinning some rope to a wall.

Before going into how we draw the smoothed line between those points, let’s examine the benefits:

- We only need four points to draw a smoothed line between the middle two. As you are drawing an ink stroke with your stylus, we are constantly able to smooth the stroke. I.e. we can do real time smoothing.
- The computation is bounded, and by some neat compiler optimizations and limiting the number of samples when drawing the smoothed line (see item 2 below) we can ensure ink smoothing won’t be on the critical path of performance.

There are a few things to keep in mind:

- We need to handle drawing a smoothed line between the first two points (P0 & P1), as well as drawing the smoothed line between the last two points on the curve. I do these by faking up those points and applying the same spline technique.
- I keep writing “draw a smoothed line between two points”. We can’t draw a smoothed line; we can only draw a bunch of straight lines that look smooth. So when I say “draw a smoothed line between two points” what I mean to say is “draw many straight lines that look smooth which connect two points”. We just sample points along the curved line at regular intervals which are known to look smooth at the pixel level.

**Cubic Spline & Cardinal Spline**

Now on to the mathematical meat… When a graphics person says that a line is smooth at a given point, what they are saying is that the line is contiguous at that point, the first derivative of the line is contiguous at that point, and the second derivative is contiguous at that point. Apologies if I’m bringing back horrible memories of high school or college calculus.

Here’s a visual of five points with the smoothed line already drawn in blue.

We can define each segment of the smoothed blue curve as being parameterized by a parameter “t” which goes from 0 to 1. So the blue line is the concatenation of 4 curves given by:

P01(t) where t ranges from 0 to 1 for the first segment (from P0 to P1)

P12(t) where t ranges from 0 to 1 for the second segment (from P1 to P2)

… etc …

Using the ` character to mean derivative, applying the definition of smooth at the endpoints of each of the segments yields a bunch of equations:

P01(t=1) = P12(t=0) P`01(t=1) = P`12(t=0) P“01(t=1) = P“12(t=0)

P12(t=1) = P23(t=0) P`12(t=1) = P`23(t=0) P“12(t=1) = P“23(t=0)

… etc …

Solving those equations *exactly* is trying. See spline interpolation. In general, if you are looking for a polynomial to satisfy an equation with second derivatives, you are shopping for a polynomial of degree 3, aka a cubic polynomial. Hence the ‘cubic’ in cubic spline.

The Wikipedia page shows a solution to fit the smoothness equations, but a lot of work has been done in this space to come up with a *more computationally feasible* solution that* looks just as smooth*. Basically, we lessen the second derivative equations and say P“01(t=1) ~= P“12(t=0), etc. This opens up many possibilities – look up any cubic spline and you’ll see many options.

After much experimenting, I found that the Cardinal spline works best for our ink strokes. The cardinal spline solution for the smoothed curve between 4 points P0, P1, P2, P3 is as follows:

The factor L is used to simulate the “tension in the heavy rope”, and can be tuned as you see fit. We chose a value around 0.5. If you are so inclined, you can also write out P23(t), take a bunch of derivatives and see this fits the smoothness equations. If you are a high school calculus teacher, please don’t make your students do this for homework.

The formula can be expressed in C++:

for (int i=0; i<numPoints; i++)

{

float t = (float)i/(float)(numPoints-1);

smoothedPoints_X[i] = (2*t*t*t – 3*t*t + 1) * p2x

+ (-2*t*t*t + 3*t*t) * p3x

+ (t*t*t – 2*t*t + t) * L*(p3x-p1x)

+ (t*t*t – t*t) * L*(p4x-p2x);

smoothedPoints_Y[i] = (2*t*t*t – 3*t*t + 1) * p2y

+ (-2*t*t*t + 3*t*t) * p3y

+ (t*t*t – 2*t*t + t) * L*(p3y-p1y)

+ (t*t*t – t*t) * L*(p4y-p2y);

}

numPoints (the number of points to sample on our smoothed line) is based on the minimum interval for what we thought looked good.

**Performance**

Like I mentioned before, we do real-time ink smoothing. That is to say an ink stroke is smoothed as it is drawn. We need to make sure that drawing a smooth line does not take too long otherwise we’ll notice a drop in frame rate where the ink stroke lags behind your stylus.

One of the benefits of writing this app in C++ is the opportunity for compiler optimizations to kick in. In this particular case, the cardinal spline equations are auto-vectorized by the Visual Studio 2012 C++ compiler. This yields a **30% performance boost** when smoothing ink strokes, ensuring we can smooth ink points as fast as Windows can sample them. Also, any extra computing time saved lets us (a) do more computations to make the app better, or (b) finish our computations early, putting the app to sleep thus saving power.

Read all about the auto vectorizer here: http://blogs.msdn.com/b/nativeconcurrency/archive/2012/04/12/auto-vectorizer-in-visual-studio-11-overview.aspx

The post Project Austin Part 3 of 6: Ink Smoothing appeared first on C++ Team Blog.

]]>The post Project Austin Part 2 of 6: Page Curling appeared first on C++ Team Blog.

]]>In this blog post, I’ll explain how we implemented page turning in the “Full page” viewing mode. We wanted to make flipping through the pages in Austin to feel like flipping through pages in a real book. To that end, we built on some existing published work to achieve *performant* and *realistic* page curling.

Before going further, take a look at a video of page curling in action!

(you can download the video in mp4 format using this link)

**Realistic page curling**

A brilliant paper by Hong et. al. called “Turning Pages of 3D Electronic Books” claims that turning a page of a physical book can be simulated as deforming a page around a cone. See [1] for the details.

Here’s a (poorly drawn) diagram to help explain the concept in the paper. The flat sheet of paper is deformed around the cone to simulate curling. By changing the shape and position of the cone you can simulate more or less curling.

Similarly, you can also curl a flat sheet of paper around a cylinder. Here’s another (poorly drawn) diagram to help explain that concept.

To simulate curling, we use a combination of curling around a cone and curling around a cylinder:

- If the user is trying to curl from the top-right of the page, we simulate pinching the top right corner of a piece of paper by deforming around a
*cone*. - If the user is trying to curl from the center-right of the page, we simulate pinching the center of a piece of paper by deforming around a
*cylinder*. - If the user is trying to curl from the bottom-right of the page, we simulate pinching the bottom right corner of a piece of paper by deforming around the
*cone flipped upside down*.

Anywhere in between and we use a combination of cone & cylinder deforming.

**Some geometry**

Here are the details to transform a page around a cylinder. There is similar geometry to transform a page to a cylinder described in [1]. Given the point P_{flat} with coordinates {x_{1}, y_{1}, z_{1} = 0} of a flat page, we want to transform it into P_{curl} with coordinates {x_{2}, y_{2}, z_{2}} the point on a cylinder with radius r that is lying on the ‘spine’ of the book. Consider the following diagram. Note the x & z axes (the y axis is in & out of your computer screen). Also keep in mind I am representing the flat paper & cylinder using the same colors as in the diagrams above.

The key insight is that the distance from the origin to P_{flat} (x_{1}) is the same arc distance as from the origin to P_{curl} *along the cylinder*. Then, from simple geometry, we can say that β = x_{1} / r. Now, to get P_{curl}, we take the origin, move it down by ‘r’ on the z axis, rotate about β, then move it up by ‘r’ on the z axis. So, the math ends up being:

The above equations compute P_{curl} by wrapping a flat page around cylinder. [1] contains the equations to compute a different P_{curl} by wrapping a flat page around a cone. Once we compute both P_{curl }values, we combine the results based on where the user is trying to curl the page. Lastly, after we have computed the two curled points, we rotate the entire page about the spine of the book.

The specific parameters are tuned by hand: the cone parameters, the cylinder width, and the rotation about the spine of the book.

**Code**

The source code for Austin, including the bits specific to page curling described here, can be downloaded on CodePlex. The page curling transformation is done in journal/views/page_curl.cpp, specifically in page_curl::curlPage(). The rest of the code in that file is to handle uncurling pages (forwards or backwards) when the user lifts their finger off the screen. I’m omitting some important details, but this code gives the rough idea.

for (b::int32 j = 0; j < jMax; j++)

{

…

for (b::int32 i = 0; i < iMax; i++)

{

{load up x, y, z=0}

float coneX = x;

float coneY = y;

float coneZ = z;

{

// Compute conical parameters coneX, coneY, coneZ

…

}

float cylX = x;

float cylY = y;

float cylZ = z;

{

float beta = cylX / cylRadius;

// Rotate (0,0,0) by beta around line given by x = 0, z = cylRadius.

// aka Rotate (0,0,-cylRadius) by beta, then add cylRadius back to z coordinate

cylZ = -cylRadius;

cylX = -cylZ * sin(beta);

cylZ = cylZ * cos(beta);

cylZ += cylRadius;

// Then rotate by angle about the y axis

cylX = cylX * cos(angle) – cylZ * sin(angle);

cylZ = cylX * sin(angle) + cylZ * cos(angle);

// Transform coordinates to the page

cylX = (cylX * pageCoordTransform) – pageMaxX;

cylY = (-cylY * pageCoordTransform) + pageMaxY;

cylZ = cylZ * pageCoordTransform;

}

// combine cone & cylinder systems

x = conicContribution * coneX + (1-conicContribution) * cylX;

y = conicContribution * coneY + (1-conicContribution) * cylY;

z = conicContribution * coneZ + (1-conicContribution) * cylZ;

vertexBuffer[jOffset + i].position.x = x;

vertexBuffer[jOffset + i].position.y = y;

vertexBuffer[jOffset + i].position.z = z;

}

}

**Automatic Vectorization**

A new feature in the Visual Studio 2012 C++ compiler is automatic vectorization. The C++ compiler analyzes loop bodies and generates code targeting the SSE2 instruction set to take advantage of CPU vector units. For an introduction to the auto vectorizer, and plenty of other information, please see the vectorizer blog series.

The inner loop above is vectorized by the Visual Studio 2012 C++ compiler. The compiler is able to vectorize all of the transcendental functions in math.h, along with the standard arithmetic operations (addition, multiplication, etc). The generated code loads four values of x, y, and z. Then it computes four values of cylX, cylY, cylZ at a time, computes curlX, curlY, curlZ at a time, and stores the result into the vertex buffer for four vertices.

I know the code gets vectorized because I specified the /Qvec-report:1 option in my project settings, under Configuration Properties -> C/C++ -> Command Line, as per the following picture:

Then, after compiling, the output window shows which loops were vectorized, as per the following picture:

*Eric’s editorial: we decided late during the product cycle to include the /Qvec-report:1 and /Qvec-report:2 switches, and we did not have time to include them in the proper menu location.*

If you do not see a loop getting vectorized and wonder why, you can specify the /Qvec-report:2 option. We offer some guidance on handling loops that are not vectorized in a vectorizer blog post.

Because of the power of CPU vector units, the ‘i’ loop gets sped up by a factor of 1.75. In this instance, we are able to compute P_{curl} (the combination of cone & cylinder) for four vertices at a time. This frees up CPU time for other rendering tasks, such as shading the page.

**Performance**

To curl a single page, we need to calculate P_{curl} for each vertex comprising a piece of paper. To my count, this involves 4 calls to sin, 3 calls to cos, 1 arcsin, 1 sqrt, and a dozen or so multiplications, additions and subtractions – for each vertex in a piece of paper – for each frame that we are rendering!

We aim to render at 60fps, which means we have around 15 milliseconds to curl the pages vertices and render them — otherwise the app will feel sluggish. With this loop getting auto vectorized, we’re able to free up CPU time for other rendering tasks, such as shading the page.

**References**

[1] L. Hong, S.K. Card, and J. Chen, “Turning Pages of 3D Electronic Books”, in Proc. 3DUI, 2006, pp.159-165.

The post Project Austin Part 2 of 6: Page Curling appeared first on C++ Team Blog.

]]>