Writing a custom Holographic Remoting Player app

If you're new to Holographic Remoting, you may want to read our overview.

By creating a custom Holographic Remoting player app, you can create a custom application capable of displaying immersive views from on a remote machine on your HoloLens 2. All code on this page and working projects can be found in the Holographic Remoting samples github repository.

A Holographic Remoting player lets your app display holographic content rendered on a desktop PC or UWP device like the Xbox One with access to more system resources. A Holographic Remoting player app streams input data to a Holographic Remoting remote application and receives back an immersive view as video and audio stream. The connection is made using standard Wi-Fi. To create a player app, use a NuGet package to add Holographic Remoting to your UWP app. Then write code to handle the connection and to display an immersive view.

Prerequisites

A good starting point is a working DirectX based UWP app that already targets the Windows Mixed Reality API. For details see DirectX development overview. If you don't have an existing app and want to start from scratch the C++ holographic project template is a good starting point.

Get the Holographic Remoting NuGet package

The following steps are required to add the NuGet package to a project in Visual Studio.

1. Open the project in Visual Studio.
2. Right-click the project node and select Manage NuGet Packages...
3. In the panel that appears, select Browse and then search for "Holographic Remoting".
4. Select Microsoft.Holographic.Remoting, ensure to pick the latest 2.x.x version and select Install.
5. If the Preview dialog appears, select OK.
6. Select I Accept when the license agreement dialog appears.

Modify the Package.appxmanifest of the application

To make the application aware of the Microsoft.Holographic.AppRemoting.dll added by the NuGet package, the following steps need to be taken on the project:

1. In the Solution Explorer, right-click the Package.appxmanifest file and select Open With...
2. Select XML (Text) Editor and select OK
3. Add the following lines to the file and save

  </Capabilities>

  <!--Add lines below -->
  <Extensions>
    <Extension Category="windows.activatableClass.inProcessServer">
      <InProcessServer>
        <Path>Microsoft.Holographic.AppRemoting.dll</Path>
        <ActivatableClass ActivatableClassId="Microsoft.Holographic.AppRemoting.PlayerContext" ThreadingModel="both" />
      </InProcessServer>
    </Extension>
  </Extensions>
  <!--Add lines above -->

</Package>

Create the player context

As a first step the application should create a player context.

// class declaration:

#include <winrt/Microsoft.Holographic.AppRemoting.h>

...

private:
// PlayerContext used to connect with a Holographic Remoting remote app and display remotely rendered frames
winrt::Microsoft::Holographic::AppRemoting::PlayerContext m_playerContext = nullptr;

// class implementation:

// Create the player context
// IMPORTANT: This must be done before creating the HolographicSpace (or any other call to the Holographic API).
m_playerContext = winrt::Microsoft::Holographic::AppRemoting::PlayerContext::Create();

Next the HolographicSpace can be created, by calling HolographicSpace.CreateForCoreWindow.

m_holographicSpace = winrt::Windows::Graphics::Holographic::HolographicSpace::CreateForCoreWindow(window);

Connect to the remote app

Once the player app is ready for rendering content, a connection to the remote app can be established.

The connection can be established in one of the following ways:

1. The player app running on HoloLens 2 connects to the remote app.
2. The remote app connects to the player app running on HoloLens 2.

To connect from the player app to the remote app call the Connect method on the player context specifying the hostname and port. The default port is 8265.

try
{
    m_playerContext.Connect(m_hostname, m_port);
}
catch(winrt::hresult_error& e)
{
    // Failed to connect. Get an error details via e.code() and e.message()
}

Listening for incoming connections on the player app can be done by calling the Listen method. Both the handshake port and transport port can be specified during this call. The handshake port is used for the initial handshake. The data is then sent over the transport port. By default port number 8265 and 8266 are used.

try
{
    m_playerContext.Listen(L"0.0.0.0", m_port, m_port + 1);
}
catch(winrt::hresult_error& e)
{
    // Failed to listen. Get an error details via e.code() and e.message()
}

Handling connection-related events

The PlayerContext exposes three events to monitor the state of the connection

1. OnConnected: Triggered when a connection to the remote app has been successfully established.

m_onConnectedEventToken = m_playerContext.OnConnected([]() 
{
    // Handle connection successfully established
});

2. OnDisconnected: Triggered if an established connection is terminated or a connection couldn't be established.

m_onDisconnectedEventToken = m_playerContext.OnDisconnected([](ConnectionFailureReason failureReason)
{
    switch (failureReason)
    {
        // Handle connection failed or terminated.
        // See ConnectionFailureReason for possible reasons.
    }
}

3. OnListening: When listening for incoming connections starts.

m_onListeningEventToken = m_playerContext.OnListening([]()
{
    // Handle start listening for incoming connections
});

Additionally the connection state can be queried using the ConnectionState property on the player context.

winrt::Microsoft::Holographic::AppRemoting::ConnectionState state = m_playerContext.ConnectionState();

Display the remotely rendered frame

To display the remotely rendered content, call PlayerContext::BlitRemoteFrame while rendering a HolographicFrame.

BlitRemoteFrame requires that the back buffer for the current HolographicFrame is bound as render target. The back buffer can be received from the HolographicCameraRenderingParameters via the Direct3D11BackBuffer property.

When called, BlitRemoteFrame copies the latest received frame from the remote application into the BackBuffer of the HolographicFrame. Additionally the focus point set is set, if the remote application has specified a focus point during the rendering of the remote frame.

// Blit the remote frame into the backbuffer for the HolographicFrame.
winrt::Microsoft::Holographic::AppRemoting::BlitResult result = m_playerContext.BlitRemoteFrame();

On success, BlitRemoteFrame returns BlitResult::Success_Color. Otherwise it returns the failure reason:

• BlitResult::Failed_NoRemoteFrameAvailable: Failed because no remote frame is available.
• BlitResult::Failed_NoCamera: Failed because no camera present.
• BlitResult::Failed_RemoteFrameTooOld: Failed because remote frame is too old (see PlayerContext::BlitRemoteFrameTimeout property).

BlitResult can also return BlitResult::Success_Color_Depth under the following conditions:

• The remote app has committed a depth buffer via HolographicCameraRenderingParameters.CommitDirect3D11DepthBuffer.
• The custom player app has bound a valid depth buffer before calling BlitRemoteFrame.

If these conditions are met, BlitRemoteFrame will blit the remote depth into the currently bound local depth buffer. You can then render additional local content, which will have depth intersection with the remote rendered content. Additionally you can commit the local depth buffer via HolographicCameraRenderingParameters.CommitDirect3D11DepthBuffer in your custom player to have depth reprojection for remote and local rendered content.

Projection Transform Mode

One problem, which surfaces when using depth reprojection via Holographic Remoting is that the remote content can be rendered with a different projection transform than local content directly rendered by your custom player app. A common use-case is to specify different values for near and far plane (via HolographicCamera::SetNearPlaneDistance and HolographicCamera::SetFarPlaneDistance) on the player side and the remote side. In this case, it's not clear if the projection transform on the player side should reflect the remote near/far plane distances or the local ones.

Starting with version 2.1.0 you can control the projection transform mode via PlayerContext::ProjectionTransformConfig. Supported values are:

• Local - HolographicCameraPose::ProjectionTransform returns a projection transform, which reflects the near/far plane distances set by your custom player app on the HolographicCamera.
• Remote - Projection transform reflects the near/far plane distances specified by the remote app.
• Merged - Near/Far plane distances from your remote app and your custom player app are merged. By default this is done by taking the minimum of the near plane distances and the maximum of the far plane distances. In case either the remote or local side are inverted, say far < near, the remote near/far plane distances are flipped.

Optional: Set BlitRemoteFrameTimeout

The PlayerContext::BlitRemoteFrameTimeout property specifies the amount of time a remote frame is reused if no new remote frame is received.

A common use-case is to enable the BlitRemoteFrame timeout to display a blank screen if no new frames are received for a certain amount of time. When enabled the return type of the BlitRemoteFrame method can also be used to switch to a locally rendered fallback content.

To enable the timeout, set the property value to a duration equal or greater than 100 ms. To disable the timeout, set the property to zero duration. If the timeout is enabled and no remote frame is received for the set duration, BlitRemoteFrame will fail and return Failed_RemoteFrameTooOld until a new remote frame is received.

using namespace std::chrono_literals;

// Set the BlitRemoteFrame timeout to 0.5s
m_playerContext.BlitRemoteFrameTimeout(500ms);

Optional: Get statistics about the last remote frame

To diagnose performance or network issues, statistics about the last remote frame can be retrieved via the PlayerContext::LastFrameStatistics property. Statistics are updated during the call to HolographicFrame::PresentUsingCurrentPrediction.

// Get statistics for the last presented frame.
winrt::Microsoft::Holographic::AppRemoting::PlayerFrameStatistics statistics = m_playerContext.LastFrameStatistics();

For more information, see the PlayerFrameStatistics documentation in the Microsoft.Holographic.AppRemoting.idl file.

Optional: Custom data channels

Custom data channels can be used to send user data over the already-established remoting connection. For more information, see custom data channels.

Optional: Over-Rendering

Holographic Remoting predicts where the user's head will be at the time the rendered images appear on the displays. However, this prediction is an approximation. Therefore, the predicted viewport on the remote app and the later actual viewport on the player app can differ. Stronger deviations (for example, due to unpredictable motion) could cause black regions at the borders of the viewing frustum. Starting with version 2.6.0 you can use Over-Rendering to reduce the black regions and enhance the visual quality by artificially increasing the viewport beyond the viewing frustum.

Over-Rendering can be enabled via PlayerContext::ConfigureOverRendering.

The OverRenderingConfig specifies a fractional size increase to the actual viewport, so that the predicted viewport becomes larger and less cutting occurs. With an increased viewport size, the pixel density decreases, so the OverRenderingConfig allows you to increase the resolution as well. If the viewport increase is equal to the resolution increase the pixel density remains the same. OverRenderingConfig is defined as:

struct OverRenderingConfig
{
    float HorizontalViewportIncrease; // The fractional horizontal viewport increase. (e.g. 10% -> 0.1).
    float VerticalViewportIncrease; // The fractional vertical viewport increase. (e.g. 10% -> 0.1).
                
    float HorizontalResolutionIncrease; // The fractional horizontal resolution increase. (e.g. 10% -> 0.1).
    float VerticalResolutionIncrease; // The fractional vertical resolution increase. (e.g. 10% -> 0.1).
};

Optional: Coordinate System Synchronization

Starting with version 2.7.0 coordinate system synchronization can be used to align spatial data between the player and remote app. For more information, see Coordinate System Synchronization with Holographic Remoting Overview.

See Also

• Holographic Remoting Overview
• Writing a Holographic Remoting remote app using Windows Mixed Reality APIs
• Writing a Holographic Remoting remote app using OpenXR APIs
• Custom Holographic Remoting data channels
• Establishing a secure connection with Holographic Remoting
• Holographic Remoting troubleshooting and limitations
• Holographic Remoting software license terms
• Microsoft Privacy Statement