DirectShow Playback and Latency Test (Compact 7)

Article
03/14/2014

3/12/2014

The DirectShow Playback and Latency Tests validate the ability to run the DirectShow Performance Tests and provide latency measurements. These tests check for graph build latency and that the correct filters have been added to the graph. These tests fall into two categories:

*Build and playback testing: Ensure that you can create a graph for a particular media clip. Before you run the DirectShow Performance Tests, you must run and pass these tests. If you cannot build a graph for your clip and play the file from beginning to end in a reasonable time period then, then these tests as well as the performance tests do not apply to you.

*Latency testing: Provide latency measurements from playing a clip from when a frame leaves the decoder until it is displayed, as well as during state changes (play/pause/stop). These are performance metrics that help you build your device and software. Again, these tests will not work if a clip cannot pass the build and playback tests above.

These tests do not provide media. The XML files use a placeholder filename which must be modified to play the files you provide. You should provide media appropriate to your device and sufficient to thoroughly test it.

Test Prerequisites

Your device must meet the following requirements before you run this test.

The following table shows the hardware requirements for the DirectShow Playback and Latency Tests.

Requirement	Description
Windows Embedded Compact-based device with network card and drivers	The test device must have network access
Audio and video card, as needed	Depends on the device capabilities
Network equipment	Hubs/routers/switches, cables, and so forth
Server with content	Provides content for testing
Hard disk, if needed for local media playback	May have some other storage device, as well

The following table shows the software requirements for the DirectShow Playback and Latency Tests.

Requirement	Description
Tux.exe	Test harness, required for testing
Kato.dll	Logging engine, required for logging data
Graphtest.dll	Test library
stream_build_test.xml	Sample xml file is provided; modify as needed
Media content	Files you wish to play back
Audio, video, and network drivers	As needed for the test device
Quartz.dll	Library required for DirectShow operation

The following table shows the SYSGENs that you may need to build into your operating system.

SYSGEN	Condition
BSP_NOSHAREETH	Required
SYSGEN_DSHOW
SYSGEN_DSHOW_WMT
SYSGEN_DSHOW_DMO
SYSGEN_DSHOW_DISPLAY
SYSGEN_DSHOW_HTTPSTREAMER	Required for DirectShow, along with Quartz.dll, as mentioned above
SYSGEN_DDRAW	Needed if you want the video renderer to run in DirectDraw; video drivers and cards must also support DirectDraw
SYSGEN_DSHOW_WMT_HTTP	Needed to run HTTP tests
SYSGEN_ATAPI (or other driver, as appropriate)	Hard disk driver, needed for streaming from a hard disk. If you want to stream from another kind of storage device, you must include the SYSGEN with the appropriate drivers.
SYSGEN_DSHOW_WMT_LOCAL	Needed to run local storage tests in all cases where you use a storage device
SYSGEN_DSHOW_MMS	Needed to run MMS tests
SYSGEN_DSHOW_WMV	Needed to test wmv content
SYSGEN_DSHOW_WMA	Needed to test wma content
SYSGEN_DSHOW_MP3
SYSGEN_DSHOW_MPEGA
SYSGEN_DSHOW_MPEGV
SYSGEN_DSHOW_MPEGSPLITTER	Needed

In addition, you must have any SYSGENs needed to include the codecs or other components for running other content of interest.

Subtests

This test has no subtests.

Setting Up the Test

The following procedure describes how to set up the test environment.

Setting up the test environment for the DirectShow Playback and Latency Tests

1. Set up a network (preferably private, to eliminate noise)

2. Set up a server with your media content, a web server for HTTP testing (for example, an ISS server), and a Windows Media server for MMS and intelligent HTTP streaming, as required for your particular testing needs.

3. Ensure that your device has access to the server and its content.

4. Modify the IP address and server name in the XML file to point to the servers.

5. Modify the <Media></Media> tags n the SML file to point to valid media. By default, this points to a dummy location.

Running the Test

Creating and Modifying the XML File for the DirectShow Playback and Latency Tests

A media XML file can be found under private\test\multimedia\directx\dshow\playback\tests\XMLSuites\stream_build_test.xml. You can update the file provided with your correct server paths, or create your own and update the command line to point to your XML file; see Command Line Parameters for the DirectShow Playback and Latency Tests.

Add or remove tests from the XML suite to change the test cases that run. You can also update parameters for the tests using the XML file, as described below. It is good practice to change the XML file as part of the source enlistment; then when you rebuild your source directory, the XML file in the source will overwrite the XML file on the device or release directory.

The XML schema is as follows.

copy code:

<?xml version="1.0" encoding="utf-8" ?>

<Media>

<IIS_HTTP></IIS_HTTP>

<WMS_HTTP></WMS_HTTP>

</BaseUrl>

</Media>

</MediaList>

</Verify>

</Test>

</TestGroup>

</TestList>

</TestConfig>

The root node of the XML config file contains the TestConfig node. The TestConfig node contains a media list node MediaList and a test list node TestList. The media list can be wholly contained within the config file or can refer to a different file which contains the actual list of media. The test list can be organized into Test and TestGroup nodes. Test nodes contain the name of the test to be run, the media to run on and the parameters for the test. TestGroup nodes serve to logically group together Test nodes. They also serve to specify the test id base for the group of test nodes which then get numbered sequentially following the test id base.

The following table lists the XML nodes, starting with the root node, and describes what each node can contain.

XML Node	Schema
TestConfig - the root node of the config file which contains a optional media list and a test list	<TestConfig optional:GenerateId="true">Optional:<MediaList>… </MediaList>Mandatory:<TestList>… </TestList></TestConfig> Attribute: GenerateId="true" - enables automatic generation of test ids. All other attributes are ignored
MediaList - lists the media to be used or contains a pointer to a separate file containing the media list.	<MediaList Container="filename"> The Container attribute is optional. If it is not present then at least one media element must be present<Media>… </Media> </MediaList>Attribute:Container="filepath", the path and filename
Media - gives detail about the media clip	<Media><Name>friendly name to be referred in the xml file</Name> <FileName>name of the actual file</FileName><Description>description of clip(informative only)</Description><BaseUrl><IIS_HTTP>http path of file on IIS server</IIS_HTTP> <WMS_HTTP>http path of file on WMS server</WMS_HTTP> <MMSU>mms path of file</MMSU> <release>release path</release> <Disk>path of file on storage media </Disk></BaseUrl> </Media>
TestList - lists the tests to be run.	<TestList>Contains 0 or more TestGroup or Test nodes</TestList>
TestGroup - allows tests to be logically grouped and also allows specifying test base id	<TestGroup Name="groupname" TestIdBase="base">Contains 0 or more Test nodes </TestGroup> Attribute:TestIdBase contains the base id for the test group. This is used if GenerateId is set to "true" in TestConfig and all the tests in the TestGroup node will be numbered sequentially starting from the base id.
Verify - lists the verifiers to be used	<Verify>Contains 0 or more verifier nodes</Verify> These verifier nodes are specified in the table following this one.
TestId - lists test ids to be used if automatic generation is not enabled.	<TestID>DWORD</TestID>
Media - lists the (1 or more) media+protocol to be used	<Media>medianame:protocol,medianame:protocol,…<Media>Medianame is the name given in the <Name> tag of the <Media> object.
DownloadTo - specifies the location where the media will be staged before being used	<DownloadTo>directory path</DownloadTo>
PositionList - lists the positions (as % of duration) to be used	<PositionList>pos1,pos2,…,posN</PositionsList> .Each posN is a number between 0:100 that represents a position in the media as a percentage of the duration. For instance, if the duration is 10 seconds, then pos=30 represents 3 seconds.
FilterList -lists the filter names	<FilterList>NSSOURCE:…:VIDREND</FilterList>The filter names are actually the "wellknownname' of the filter to the test. The only filters accepted are the ones specified in the test binary. Currently accepted values: WMADEC, WMVDEC, MP43DEC, MPGDEC, MP3DEC, MP13DECDMO, MP13DEC, MP2ADEC, MP2VDEC, MP2VDEC, MP2VENC, AVISPLIT, WAVPAR, AVISPLIT, MPGSPLIT, MPEG2DMUX, MPEG2Demux, AUDREND, VIDREND, ICMDISP, ACMDISP, NSSOURCE, ASYNCRDR, URLRDR, CLRCONV, AVIMUX, MPEGMUX, OVMIXER, FILEWR. For a full list of these filters and details, look at the map structure at line 122 of Playback\Framework\FilterDesc.cpp in the test source code.
StateChangeSequece - specifies the state change pattern and number of state changes	<StateChangeSequence>InitialState,Sequence,NumStateChanges,DelayBetweenChangesInMilliseconds</StateChangeSequence>InitialState can be Stopped, Paused, Running. Sequence can be PlayPause, PlayStop, PauseStop, RandomSequence.
Test - specifies the actual test to be run and the parameters.	<Test Name="testname" Desc="desc">params</Test>.Attributes:Name specifies the test function to be run. This string is associated with the test function in an internal function table.Desc is concatenated to the generic description of the test when listing out tests.The <Test> tag contains parameters specific to the type of test description expected for this test. These parameters are specified in a table below.

The following table shows the parameters expected for the <Test> tag.

Parameter expected	Description
Playback test parameters	Contains nodes in the following order. Optional nodes can be omitted, but the order must be maintained. Optional: <TestId>DWORD</TestId>Mandatory: <Media>one or more entries of the form "medianame:protocol" separated by ‘,'</Media>Optional: <DownloadTo>download path</DownloadTo>Optional: <PositionList>start position,stop position</PositionList>Optional: <Verify>…</Verify>
Build test parameters	Contains nodes in the following order. Optional nodes can be omitted, but the order must be maintained. Optional: <TestId>DWORD</TestId>Mandatory: <Media>one or more entries of the form "medianame:protocol" separated by ‘,'</Media>Optional: <DownloadTo>download path</DownloadTo>Optional: <FilterList> list of filters, separated by ‘:' </FilterList>Optional: <Verify>…</Verify>
State change test parameters	Contains nodes in the following order. Optional nodes can be omitted, but the order must be maintained.Optional: <TestId>DWORD</TestId>Mandatory: <Media>one or more entries of the form "medianame:protocol" separated by ‘,'</Media>Optional: <DownloadTo>download path</DownloadTo>Mandatory: <StateChangeSequence>…</StateChangeSequence>Optional: <Verify>…</Verify>

Command Line Parameters for the DirectShow Playback and Latency Tests

The following table shows the command line parameter for the DirectShow Playback and Latency Tests.

Command line parameter	Description
/Config filename	Specifies the filename, and optionally the path, of the xml file to be used in running the test. If no path is provided with the filename, the test will first try to open the file from the \Windows folder and then from the current folder the test is running from.

Verifying the Test

When the test completes running, verify that "PASS" appears in the test log for all of the tests listed in the configuration file.

Troubleshooting the Test

Check that the XML file is configured as desired. The default values in the XML file will usually fail. This is especially true of the latency and duration thresholds, which are set to arbitrary values. Update the XML file to reflect the desired latency. The test will fail if the latency experienced is larger than the threshold specified in the XML file.

In general, the tests may fail if:

* the URL is inaccessible, or if the xml file points to an invalid URL

* the server containing the media is not accessible

* the exact URL on the server is inaccessible

* the network is inaccessible or the proxy or firewall are configured incorrectly

* the correct SYSGENs needed to pull in the necessary DirectShow components are missing

* you do not have or cannot initialize COM

* the test device lacks a storage device, or the XML file is not pointing to the storage device or the media clip

* the test device has run out of memory

* the command line points to the wrong XML file

The following table shows common reasons for failure for each test case.

Test case	Typical reasons for failure
0	DirectShow cannot find an appropriate source filter for the URL.
1	A filter in the <FilterList> cannot be loaded. Cannot create an end-to-end filter graph of the URL.
2	DirectShow cannot find an appropriate source filter for the URL. Cannot render the unconnected pins all the way to the renderer.
3	Xml file does not specify a source filter, or the source filter is unknown. Xml file does not specify a renderer filter, or the renderer filter is unknown. The source filter or the renderer specified cannot be loaded. Cannot connect the pins of the source filter, through various transform filters if needed, to the renderer specified.
4	The URL cannot be rendered and the test cannot create an end-to-end, completely connected graph. All the filters in the <CorrectGraph> are not present in the connected graph.
5	The URL cannot be rendered and the test cannot create an end-to-end, completely connected graph. Any of the essential interfaces are not returned.
100	The URL cannot be rendered and the test cannot create an end-to-end, completely connected graph. Cannot set the playback position to play to and from. Cannot run the graph successfully. It takes longer to playback than expect (calculated based of off clip duration). Tester clicks No on the message box asking to verify playback.
101	Cannot copy the clip locally (URLMON is used to do this); failure could be due to: 1) Lack of space on the device in the location specified; 2) The location specified in the <DownloadTo> tag does not exist; 3) The clip is too big to fit in the URLMON cache. The URL cannot be rendered and the test cannot create an end-to-end, completely connected graph. Cannot set the playback position to play to and from. Cannot run the graph successfully. It takes longer to playback than expected (calculated based of off clip duration). Tester clicks No on the message box asking to verify playback.
102	The URL cannot be rendered and the test cannot create an end-to-end, completely connected graph. Cannot set the playback position to play to and from. Cannot run the graph successfully. It takes longer to playback than expected (calculated based of off clip duration). The playback duration is not within the threshold specified. The threshold can be specified in the <PlaybackDuration> tag as a comma separated list. First number is the percent deviation allowed on total playback time. Second number is the deviation allowed in ms. If the difference between expected playback and actual playback is not covered by either of these two thresholds, then the test fails.
200 201	The URL cannot be rendered and the test cannot create an end-to-end, completely connected graph. Cannot set the playback position to play to and from. Cannot run the graph successfully. It takes longer to playback than expected (calculated based of off clip duration). The startup latency is not within the specified threshold. Can configure this threshold via the <StartupLatency></StartupLatency> tag. Specify a latency in ms. Currently this is set at 300 ms.
300 301 302	The URL cannot be rendered and the test cannot create an end-to-end, completely connected graph. Cannot run the graph successfully. Cannot change states. State change latency is not within expected threshold (in ms). This value can be modified via the <VerifyStateChangeLatency> tag. The value is specified in ms. Currently set at 2000 ms. Test measures the latency for every state change it does, plus calculates and prints out an average state change latency over all the runs.
303	The URL cannot be rendered and the test cannot create an end-to-end, completely connected graph. Cannot run the graph successfully. Cannot change the states successfully. Decoder run to first sample latency is not within expected threshold (in ms). This value can be modified via the <DecodedVideoLatencyRunToFirstSample> tag. The value is specified in ms. Currently set at 300 ms.

For additional platform specific issues, consult the CTK articles on the TechNet wiki.

Share via