Implement Hardware Acceleration for Graphics in Silverlight for Windows Embedded [Reference] (Compact 7)

3/12/2014

You can use hardware acceleration for the bitmaps that are generated by the rendering process in a Microsoft Silverlight for Windows Embedded application. By using hardware acceleration, you can offload common graphics operations from the main microprocessor onto a display controller by blitting bitmaps first to an off-screen graphical frame buffer, or surface, before they are rendered on the screen. This process creates a faster and more stable graphics environment. If you do not use hardware acceleration, Windows Embedded Compact 7 uses Graphics Device Interface (GDI) to draw UI objects pixel-by-pixel onto the primary display surface in the order they were first loaded into the object tree during XAML parsing.

Hardware acceleration relies on a process called cached composition, which minimizes CPU usage during rasterization by using the graphics processing unit (GPU) and memory instead of the CPU. The disadvantage of this technique is that it requires additional video and system memory on the hardware board.

Graphics hardware acceleration in Silverlight for Windows Embedded is based on one of the following:

• DirectDraw   Supports hardware-accelerated two dimensional graphics. For more information, see DirectDraw in the Windows Embedded Compact 7 Help.

• OpenGL   Supports hardware-accelerated two-dimensional or three-dimensional graphics, skewing, rotation, and plane projection. For more information, see Introduction to OpenGL on MSDN.

  Note

  If you implement hardware acceleration based on OpenGL, the XAML UI that is rendered on the screen has a maximum size limitation of 2,048 × 2,048 pixels.

This tutorial contains the following major steps.

• Step 1: Add Support for Hardware Acceleration to the OS Design
  Use this procedure to add support for hardware acceleration to your OS design by including BSP flags. If you are not going to customize the rendering plug-ins, you can proceed to Step 6.

• Step 2: Customize the Hardware Configuration and Graphics-Rendering Behavior
  Use this procedure to learn how to customize the graphics renderer that Silverlight for Windows Embedded uses when it displays graphics. You do not have to customize the graphics renderer unless you want to maximize the performance of the renderer code on your specific device hardware.

• Step 3: Customize the DrawTriangleStrip Method
  Use this procedure to learn how to customize the DrawTriangleStrip method. DrawTriangleStrip is called internally to draw a series of triangles by using a set of input vertices that are provided in an array of XRVertex structures. DrawTriangleStrip is part of the rendering plug-in.

• Step 4: Build Your Code
  If you customized the rendering plug-in, use this procedure to build the DLL.

• Step 5: Include the Security Model
  Use this procedure to help ensure that access to your customized DLL is secure by including the trusted security model and implementing load privilege for XamlRenderPlugin.dll.

• Step 6: Build the OS Design into a Run-Time Image
  Use this procedure to create a run-time image that includes support for hardware acceleration and your customized DLL if you created one.

• Step 7: Use Hardware Acceleration in Your Application
  Use this procedure to learn how to cache bitmaps and use the z-order of UI elements to optimize hardware acceleration.

Prerequisites

Before you begin this tutorial, be sure that:

• You have a Silverlight for Windows Embedded application. For more information, see Getting Started with Silverlight for Windows Embedded.
• Your hardware meets the guidelines that are described in Silverlight for Windows Embedded Hardware Recommendations.

Step 1: Add Support for Hardware Acceleration to the OS Design

You support hardware acceleration in an OS design by adding a board support package (BPS) flag. For example, after you add the BSP flag for DirectDraw, then you can customize the DirectDraw plug-in (see Step 2: Customize the Hardware Configuration and Graphics-Rendering Behavior through Step 7: Use Hardware Acceleration in Your Application below), or just use hardware acceleration in your application (see Step 6: Build the OS Design into a Run-Time Image and Step 7: Use Hardware Acceleration in Your Application below).

To set a BSP flag at the command line

1. In Platform Builder, on the Build menu, click Open Release Directory in Build Window.

2. To see if one of the plug-ins is already set, type Set | findstr BSP_XRPLUGIN. If nothing is returned, you can proceed with the next step.

3. At the command prompt, type Set, type a space, and then type the flag you want to set. For example:

   • To add the DirectDraw rendering plug-in to your OS design, type Set BSP_XRPLUGIN_DDRAW=1.
   • To add the OpenGL rendering plug-in to your OS design, type Set BSP_XRPLUGIN_OPENGL=1.

If you are not going to customize the rendering plug-in you chose, proceed to Step 6.

Step 2: Customize the Hardware Configuration and Graphics-Rendering Behavior

You can customize the graphics renderer that is used internally by Silverlight for Windows Embedded when it displays graphics. You do not have to customize the graphics renderer unless you want to maximize the performance of the renderer code on your specific device hardware.

The default DirectDraw plug-in is implemented in:

• IRenderer   To customize, implement your changes in %_WINCEROOT%\Public\Common\Oak\Xamlrendererplugin\Ddraw\Ddrawrenderer.cpp.
• ICustomSurface   To customize, implement your changes in %_WINCEROOT%\Public\Common\Oak\Xamlrendererplugin\Ddraw\Ddrawsurface.cpp.
• ICustomGraphicsDevice   To customize, implement your changes in %_WINCEROOT%\Public\Common\Oak\Xamlrendererplugin\Ddraw\Ddrawdevice.cpp.

The default OpenGL plug-in is implemented in:

• IRenderer   To customize, implement your changes in %_WINCEROOT%\Public\Common\Oak\Xamlrendererplugin\Opengl\Openglrenderer.cpp.
• ICustomSurface   To customize, implement your changes in %_WINCEROOT%\Public\Common\Oak\Xamlrendererplugin\Opengl\Openglsurface.cpp.
• ICustomGraphicsDevice   To customize, implement your changes in %_WINCEROOT%\Public\Common\Oak\Xamlrendererplugin\Opengl\Opengldevice.cpp.

To customize the DirectDraw or OpenGL graphics renderer

1. In Platform Builder, open your OS design project.

2. In Solution Explorer, browse to <OS Design Name>\ C:\WINCE700\Public\Common\Oak\XamlRundererPlugin.

3. Expand either DDraw or OpenGL, depending on which plug-in you enabled in Step 1.

4. Browse to the .cpp file that you want to customize, and double-click the .cpp file. The files may include comments that help identify where code can be customized. For example, in %_WINCEROOT%\Public\Common\Oak\Xamlrendererplugin\Opengl\Opengldevice.cpp, there is the following comment in the COpenGLDevice::InitializeState() function:

       // Initialize the shaders
       // If we can't load binary shaders that match this hardware,
       // the following source will be compiled instead.
       // But you should really provide optimized binary shaders.
   
       CHR(InitializeShader(GL_VERTEX_SHADER, L"DefaultEffect.vs", 
       ...
       CHR(InitializeShader(GL_FRAGMENT_SHADER, L"DefaultEffect.fs", 
       ...
           "   // This is worth removing if you know your hardware does BGRA;\r\n"
           "   // even the if() test can be expensive.\r\n"
       ...
   

5. If you are customizing the OpenGL renderer, consider defining the following flags to include the related features:

   • USE_ANITIALIASING   Antialiasing refers to the smoothing of jagged edges of drawn graphics and text to improve their appearance or readability.
   • USE_DEPTH_BUFFER   A depth buffer contains per-pixel floating-point data for the z-depth of each pixel rendered. When a pixel is rendered, color data in addition to depth data can be stored. If a pixel is rendered a second time--such as when two objects overlap-depth testing determines which pixel is closer to the camera and, thus, which pixel value will be stored and drawn.
   • USE_BACKFACE_CULLING   A backface is a triangle whose normal faces away from the camera. For example, if you render a cube, there is no need to render all of the triangles that are on the back side of the cube because they aren't visible to the user.

6. After you finish customizing the code in the .cpp files, save your changes.

To customize the source code for the interfaces, you can add code to the following default method implementations, which are called internally by Silverlight for Windows Embedded when it displays graphics.

IRenderer

ICustomSurface

ICustomGraphicsDevice

Step 3: Customize the DrawTriangleStrip Method

The renderer calls the DrawTriangleStrip method to draw a series of triangles by using a set of input vertices that are provided in an array of XRVertex structures. To customize the DrawTriangleStrip method, modify one of the following files:

• For OpenGL, the COpenGLDevice DrawTriangleStrip method is located in %_WINCEROOT%\Public\Common\Oak\Xamlrendererplugin\Opengl\Opengldevice.cpp.
• For DirectDraw, the CDDrawDevice DrawTriangleStrip method is located in %_WINCEROOT%\Public\Common\Oak\Xamlrendererplugin\Ddraw\Ddrawdevice.cpp.

An XRVertex structure defines a vertex, which is the point at which two sides of an angle intersect. The following table lists the members of the XRVertex structure.

You can assign texture coordinates directly to vertices by using the u0, v0, u1, and v1 members. A texture is a bitmap of pixel colors that give an object the appearance of texture. Texture can give visual depth to an object, such as applying the texture of sandpaper to a beige background to make it appear rough. By assigning texture coordinates directly to vertices, you can control which part of a texture is mapped onto a primitive.

A texture can be assigned to a stage so that you can define how that texture is rendered. For example, you can blend multiple textures together. With XRVertex, you can change how the textures appear in different areas by applying texture coordinates for two different texture stages.

Possible effects include changing texture filling, environment mapping, or adjusting the level of detail.

Step 4: Build Your Code

If you customized the rendering plug-in in Step 2 or Step 3, use this procedure to build the DLL.

To recompile and build the graphics renderer

1. In Solution Explorer, browse to <OS Design Name>\ C:\WINCE700\Public\Common\Oak\XamlRundererPlugin.

2. Right-click either DDraw or OpenGL and click Rebuild.

3. In the Output window, verify that the build succeeded by looking for the following output message:

   ========== Build: 1 succeeded or up-to-date, 0 failed, 0 skipped ==========
   

4. Create an updated run-time image that includes the rebuilt graphics renderer.

   1. On the Build menu, click Copy Files to Release Directory.
   2. On the Build menu, click Make Run-Time Image.
   3. In the Output window, verify that the build succeeded.

If BSP_XRPLUGIN_OPENGL is set, Windows Embedded Compact renames XRRendererOpengl.dll to XamlRenderPlugin.dll during the build process. If BSP_XRPLUGIN_DDRAW is set, Windows Embedded Compact renames XRRendererDDraw.dll to XamlRenderPlugin.dll during the build process. If none is set, no XamlRendererPlugin.dll is generated; therefore, Silverlight for Windows Embedded uses an internal GDI renderer. Customers may provide their own implementation of XamlRenderPlugin.dll.

In your DLL, you must export the following names:

• RenderPluginInitialize
• RenderPluginCleanup
• CreateRenderer

You must implement your DLL to return S_OK when Silverlight for Windows Embedded calls RenderPluginInitialize.

Step 5: Include the Security Model

To help ensure that access to the DLL is secure, include the trusted security model and implement load privilege for XamlRenderPlugin.dll.

To include the security model

1. Add the Security Loader Sysgen variable (SYSGEN_LVMOD) to the OS image. You can set this Sysgen variable at the command line during a build.

2. Add one or more certificates to the codesign store on the device.

3. Ensure that the graphics-rendering DLL is either stored in ROM or signed with a certificate that is in the codesign store on the device.

These steps help prevent a malicious user from installing an unauthentic XamlRenderPlugin.dll that has malicious software that compromises all Silverlight for Windows Embedded applications on the device.

Otherwise, do not allow non-trusted applications to be installed on the device.

Step 6: Build the OS Design into a Run-Time Image

Build the OS design project and the run-time image, and download it to the device through the connection that you have already configured. For information about running your OS image on your device or virtual CEPC, see Download the OS to a Target Device for Debugging or Develop with Virtual CEPC.

If you already have a run-time image that includes the BSP flags from Step 1, and you customized the graphics renderer, you can copy the DLL for the renderer to the release directory, which is located at %_WINCEROOT%\OSDesigns\<OS design>\RelDir.

Step 7: Use Hardware Acceleration in Your Application

In your Silverlight for Windows Embedded application, to use hardware acceleration, you must set the cache mode (CacheMode) to bitmap caching ("BitmapCache") only for UI elements that use hardware acceleration at run time. You can set cache mode in either the source XAML file or in the C++ application.

We recommend that you set the cache mode for all UI elements for which you implement transformations or animations, both of which require additional GPU processing.

To set the cache mode for a UI element

1. Do one of the following:

   • In your XAML file, in the element, add a CacheMode attribute and set its value to "BitmapCache," as shown in the following code example:

     <Image x:Name="_Globe" 
         Height="138" HorizontalAlignment="Center" Margin="0,20,90,0" VerticalAlignment="Top" Width="138"
         Source="Assets/Globe138x138.png" Stretch="Fill" RenderTransformOrigin="0.5,0.5" 
         CacheMode="BitmapCache">
     

   • In your application, locate the UI element in the visual tree by using the IXRFrameworkElement::FindName method, create an IXRBitmapCache object (using the corresponding smart pointer IXRBitmapCachePtr), and then pass the IXRBitmapCache object into IXRUIElement::SetCacheMode.

     IXRListBoxPtr pElement;
     pRoot->FindName(L"FancyListBox", &pElement);
     IXRBitmapCachePtr pCache;
     pApplication->CreateObject(&pCache);
     pElement->SetCacheMode(pCache);
     

You can reduce graphics memory consumption during graphics rendering by changing the default z-order of UI elements and by positioning all cached elements higher in the z-order. We recommend that you place UI elements that use bitmap caching, such as animations, first in the z-order. When these UI elements are rendered first, it reduces the amount of back-buffer surfaces that the system must create for other elements during the rendering process, which reduces graphics memory consumption.

To specify z-order for a UI element

1. Do one of the following:

   • In your XAML file, in the element, add a Canvas.ZIndex attribute and set its value to "BitmapCache."
   • In your application, locate the UI element in the visual tree, specify a different z-order value in the Canvas.ZIndex attached property by calling IXRDependencyObject::SetAttachedProperty(const WCHAR*, const WCHAR*, int) on a UI element.

Design Guidelines

When you design a UI to take advantage of GPU-based hardware acceleration, be sure to take advantage of the bitmap cache and element z-order as previously described The following general guidelines offer additional suggestions for getting the most from GPU-based hardware acceleration. More detailed guidelines are available in the Graphics and Performance in Silverlight for Windows Embedded section.

General Guidelines

• Use UI elements that are optimized for hardware acceleration.
  For information, see Optimize UI Elements and XAML Files for Graphics Performance.
• Limit the XAML UI to a maximum size of 2,048 × 2,048 pixels for OpenGL applications.
  OpenGL has a limitation that surfaces cannot be larger than 2,048 × 2,048. If you attempt to create an OpenGL surface that is larger than 2,048 × 2,048 pixels, it will appear as a black rectangle on the screen. Silverlight for Windows Embedded will not automatically split a large surface into multiple smaller ones, so it is important to keep this limitation in mind when designing for very large screens.
• Limit large-scale animations to transformations that can be hardware-accelerated.
  Silverlight permits many attributes to be animated, but only a few of these can be hardware-accelerated. Examples of animations that cannot be accelerated include repositioning an item by changing margins and animating gradients and colors. Performing these types of animations on a small portion of the screen may not cause any performance difficulties but, for example, animating a screen-sized radial gradient is almost certain to overtax the CPU. Note that if a UI element is changing during an animation, it is repeatedly re-rendered, negating the benefits of hardware acceleration.
• Use the Visibility or Opacity properties to efficiently hide or display elements in your UI.
  While an object is hidden (visibility = collapsed), Silverlight for Windows Embedded does not hold visual data for the object in video memory, it does not walk the tree in the render pass, and it does not do any hit-test work. When the object is visible again (visibility = visible), the entire tree content has to be drawn again (re-layout). If, and only if, the object is cached, setting opacity to zero to hide the object causes Silverlight for Windows Embedded to hold a texture for that tree in video memory without affecting the performance of the compositor. When the object is visible again (opacity > 0) it will be processed normally without redrawing the content. You will likely obtain the best results by manipulating the opacity property, but with multiple or complex trees, using the visibility property may be the best approach. Try both approaches and see which works best for you.

XAML Structure Guidelines

You can also optimize rendering by structuring the XAML to improve buffer usage. The following list shows some examples to illustrate this guideline:

• If there is a set of static elements that serve as a background, group them together and put them at the start of the XAML, so that they will all be placed in the same buffer in the GPU.
• Make sure that dynamic items aren't grouped with a set of static elements. Using our button example, if we want the button to draw attention by gradually changing its brightness, we want to make sure that it isn’t in the same buffer as the static background items. Otherwise, each brightness change would require that the entire background be rasterized and composed again.
• Don’t unnecessarily create extra buffers. Buffers use memory in both the CPU and GPU, so extra buffers must not be created without reason. In our example, _Globe is placed at the end of the XAML, and only two buffers are needed. But, if _Globe was placed earlier in the XAML, the background items would be rendered in two separate buffers, meaning that three buffers would be used.

See Also

Concepts

Hardware Acceleration in Silverlight for Windows Embedded