Environment Variables in Windows HPC Server 2008 R2

Updated: August 2011

Applies To: Microsoft HPC Pack 2008 R2, Microsoft HPC Pack 2012

Windows HPC Server 2008 R2 uses many of environment variables and supports their use as aliases for recurring values or sets of values in HPC Cluster Manager and HPC Job Manager, a Command Prompt window, HPC PowerShell, or scripts and batch files. For example, you can use environment variables to specify the path for a job’s output file:

job submit /stdout:%USERPROFILE%\%CCP_JOBID%.txt myapp.exe

Note
You can use a number of built-in environment variables to affect how a Message Passing Interface (MPI) application runs when you start the MPI application with the mpiexec command. For more information, see Environment Variables for the mpiexec Command.

In this topic:

• What’s new for environment variables in Windows HPC Server 2008 R2

• Environment variable hierarchy

• System environment variables that are set when you install HPC Pack

• Cluster-wide environment variables that are set during system deployment

• Modifying or adding cluster-wide environment variables

• Job or task environment variables for console or remote desktop sessions

• Job or task environment variable to configure task cancel signal (New in SP2)

• Task environment variables that are set by the HPC Node Manager Service

• Environment variables for diagnostic tests

What’s new for environment variables in Windows HPC Server 2008 R2

In Windows HPC Server 2008 R2, job owners can define environment variables at the job level. Job environment variables are set in the run-time environment of all tasks in the job. To set the environment variable job property, a job owner can use one of the following methods (using HPC_CREATECONSOLE as an example):

• In HPC Job Manager, in the New Job or Modify Job dialog box, click the Environment Variables tab. Click Add. In Name, type HPC_CREATECONSOLE. In Value, type True.

• In HPC PowerShell, type:

  set-hpcjob –id <yourJobID> -jobenv HPC_CREATECONSOLE=True

• In a Command-prompt window, type:

  job modify <yourJobID> /jobenv: HPC_CREATECONSOLE=True

The HPC Diagnostics Service sets job-level environment variables to coordinate diagnostic runs. You can use these variables if you create custom diagnostic tests. For more information, see Environment variables for diagnostic tests.

The table below describes the environment variables that are new in Windows HPC Server 2008 R2.

Environment variable	Description
HPC_CREATECONSOLE	Job or task variable. Set this variable to have a console session created automatically when the job is started. This can be helpful for GPGPU jobs or jobs that you want to monitor through a remote desktop session. With a value of True, the HPC Node Manager Service attempts to create a console session using the credentials of job owner. If successful, the job runs in the console session. If the console session cannot be created, the job fails. The Node Manager closes the console session at the end of the job. Important The automated create console functionality must be enabled on the cluster by the cluster administrator. For more information, see Enable HPC_CREATECONSOLE for GPGPU Jobs in Windows HPC Server 2008 R2.
CCP_SERVICEREGISTRATION_PATH	Cluster-wide variable. Specifies the default location of the Service Registration folder. When you deploy service configuration files (for SOA services) to the Service Registration folder, you can manage and edit the files centrally in HPC Cluster Manager. For more information, see Deploy and Edit the Service Configuration File. You can set this variable to change the default location of the Service Registration folder. You can specify multiple paths separated by a semicolon (;). This variable works the same way that PATHS works for cmd.exe. The central service registration folder can be any shared folder that the head node, broker nodes, and compute nodes can access and read. The cluster administrators must have full control permissions on the shared folder so that configuration files can be installed to that path.
CCP_TASK_NOTIFY	Added in Service Pack 2 of Microsoft HPC Pack 2008 R2. Job or task variable. Specifies the type of signal that the HPC Node Manager Service uses to cancel a running task. The supported values for CCP_TASK_NOTIFY are: CTRL_BREAK (default) CTRL_C By default, the HPC Node Manager Service stops a task by sending a CTRL_BREAK event to the application. You can set the CCP_TASK_NOTIFY variable for your job or task if you need your application to be stopped with a CTRL_C event instead. When a running task is canceled, the application has a preset period of time to exit gracefully (as defined by the TaskCancelGracePeriod cluster property). The application can use this time to save state information, write a log message, create or delete files, or finish the current service call. To use the grace period, the application must process the CTRL_BREAK or CTRL_C event. If the application does not process the event, the task exits immediately.

^Top of page

Environment variable hierarchy

Sometimes the same environment variable can be set at different levels. If different values are set for the same environment variable, the environment variable hierarchy determines which value is used in the run-time environment of a task.

The environment variable hierarchy used for tasks that are running on the cluster is as follows:

1. Task

2. Job

3. Cluster-wide

4. User

5. System

For example, if %TMP% is set at the job level and at the task level, the task level setting takes precedence in the run-time environment of that task.

^Top of page

System environment variables that are set when you install HPC Pack

The following system environment variables are set on the head node when you install HPC Pack 2008 R2:

^Top of page

Cluster-wide environment variables that are set during system deployment

Cluster-wide environment variables can be modified by the cluster administrator, and they are set in the environment of all tasks that are running on the cluster.

The following cluster-wide environment variables are set during system deployment, and they are rarely changed manually or used in commands.

Environment variable	Description
CCP_CLUSTER_NAME	The name of the cluster, which is the same name as the computer that serves as the head node.
CCP_MPI_NETMASK	The network mask for the network interface that the Message Passing Interface (MPI) processes use. You can specify the IP address range from which the MPI processes accept connections in the form <starting IP address>/<subnet mask for this range>. For example, a network mask of 192.168.0.1/255.255.255.0 includes all the addresses between 192.168.0.1 and 192.168.0.254. This environment variable can be temporarily changed to move application traffic from the application network to a different HPC network (for example, the private network). Performing this change might be useful when troubleshooting your application network. For example, if you want to move the application traffic to the private network, and the current configuration of the private network has IP addresses in the 10.0.x.x range, with subnet mask 255.255.0.0, the following setting changes the application traffic to run on that network: CCP_MPI_NETMASK = 10.0.0.0/255.255.0.0 Note A user can set or change the network that a specific job uses for MPI traffic by including the MPICH_NETMASK environment variable in the mpiexec arguments, for example: mpiexec -env MPICH_NETMASK 157.59.0.0/255.255.0.0 myApp.exe
WCF_NETWORKPREFIX	The prefix for the network that the Windows Communication Foundation (WCF) broker and WCF service use for internal traffic.
CCP_SERVICEREGISTRATION_PATH	Specifies the location of the Service Registration folder. By default, this is C:\Program Files\Microsoft HPC Pack 2008 R2\Data\ServiceRegistration. When you deploy service configuration files (for SOA services) to the Service Registration folder, you can manage and edit the files centrally in HPC Cluster Manager. For more information, see Deploy and Edit the Service Configuration File. You can set this variable to change the location of the Service Registration folder. You can specify multiple paths separated by a semicolon (;). This variable works the same way that PATHS works for cmd.exe. The central service registration folder can be any shared folder that the head node, broker nodes, and compute nodes can access and read. The cluster administrators must have full control permissions on the shared folder so that configuration files can be installed to that path.

^Top of page

Modifying or adding cluster-wide environment variables

Custom cluster-wide environment variables can by created or modified by the cluster administrator and are set in the run-time environment of all tasks that are running on the cluster. An administrator can create and set cluster-wide environment variables to provide default values for tasks. A job owner can override the cluster-wide default by setting the same variable name with a different value at the job or task level. Job owners can set the environment variable property for a job or task when they create or modify a job or task.

Note
Built-in HPC Server environment variables begin with CCP. To avoid potential naming conflicts, do not start your variable names with CCP.

You can use the following commands to view or set (that is, create or change) cluster-wide environment variable name and value pairs:

^Top of page

Job or task environment variables for console or remote desktop sessions

HPC Server defines a set of environment variables that job owners can use to run their jobs or tasks in a console or remote desktop session. These environment variables are not set automatically. Job owners can set these variables with the environment variable property for a job or task. When the variable is set at the job level, it applies to all tasks in the job.

A job owner might want to set these variables in the following cases:

• General-purpose computations on graphics-processor units (GPGPU) programs can run faster by using the additional processing power of the GPU. Depending on the type of GPU driver, job owners might have to run GPGPU jobs in a console session. For more information, see Enable HPC_CREATECONSOLE for GPGPU Jobs in Windows HPC Server 2008 R2.

• The program has a user interface that the job owner must interact with, and the job owner plans to use Remote Desktop for an interactive console session. For more information, see Log On to a Compute Node to Interact with Your Job in Windows HPC Server 2008 R2.

The following job or task level environment variables can be set by a job owner:

Environment Variable	Description
HPC_CREATECONSOLE	Set this variable to have a console session created automatically when the job is started. This can be helpful for GPGPU jobs or jobs that you want to monitor through a remote session. Note If you run you GPGPU jobs in a console session, do not initiate a remote desktop connection to the compute nodes. The remote desktop connection takes you out of the console session, and your GPGPU job might fail. With a value of True, the HPC Node Manager service attempts to create a console session using the credentials of job owner. Only one user per node can have a console session. If successful, the job runs in the console session. If the console session cannot be created, the job fails. The Node Manager closes the console session at the end of the job. Important The automated create console functionality must be enabled on the cluster by the cluster administrator. For more information, see Enable HPC_CREATECONSOLE for GPGPU Jobs in Windows HPC Server 2008 R2.
HPC_ATTACHTOCONSOLE	Set this variable to start a job in an existing console session. This can be helpful if you have a GPGPU program, and the GPU drivers are only loaded in a console session. With a value of True, the HPC Job Scheduler service attempts to start the job in an existing console session. If a console session exists that has the same owner as the user who submitted the job, then the job starts. Only one user per node can have a console session. If there is no console session that is owned by the job owner, then the job fails. You can run qwinsta.exe at a command prompt to see a list of sessions that are currently active on your server. With a value of Try, the task will run even if it cannot be attached to a console session.
HPC_ATTACHTOSESSION	Set this variable to start a job in an existing remote desktop session. This is helpful if you have an interactive program that you would like to connect to the session and be able to remotely view the program while it is running. With a value of True, the HPC Job Scheduler service attempts to start the job in a remote desktop session. If a remote desktop connection exists that has the same owner as the user who submitted the job, then the job starts. If there is no remote desktop session that is owned by the job owner, then the job fails. You can run qwinsta.exe at a command prompt to see a list of sessions that are currently active on your server. With a value of Try, the job runs even if it cannot be attached to a remote desktop session.

To set the environment variable job property, a job owner can use one of the following methods (using HPC_CREATECONSOLE as an example):

• In HPC Job Manager, in the New Job or Modify Job dialog box, click the Environment Variables tab. Click Add. In Name, type HPC_CREATECONSOLE. In Value, type True.

• In HPC PowerShell, type:

  set-hpcjob –id <yourJobID> -jobenv HPC_CREATECONSOLE=True

• In a Command-prompt window, type:

  job modify <yourJobID> /jobenv: HPC_CREATECONSOLE=True

^Top of page

Job or task environment variable to configure task cancel signal

Service Pack 2 of Windows HPC Server 2008 R2 includes the CCP_TASK_NOTIFY environment variable to enable job owners to configure the type of signal that should be used to cancel their running tasks.

To set the environment variable job property, a job owner can use one of the following methods:

• In HPC Job Manager, in the New Job or Modify Job dialog box, click the Environment Variables tab. Click Add. In Name, type CCP_TASK_NOTIFY. In Value, type CTRL_C.

• In HPC PowerShell, type:

  set-hpcjob –id <yourJobID> -jobenv CCP_TASK_NOTIFY=CTRL_C

• In a Command-prompt window, type:

  job modify <yourJobID> /jobenv: CCP_TASK_NOTIFY=CTRL_C

Task environment variables that are set by the HPC Node Manager Service

Task environment variables apply only to a task, and most task environment variables are set automatically by the HPC Node Manager Service of the compute node that is performing the task. This is the means by which the standard task options are communicated to the process or processes. Task environment variables can be useful as aliases in commands because they are dynamic, and they contain information that is not necessarily known beforehand.

The following task environment variables are set automatically by the HPC Node Manager Service.

Environment Variable	Description
CCP_NODES	The list of nodes and number of cores per node that are allocated to the task. Nodes and cores are listed in the format numNodes<space>nodeName<space>numCores<space>. For MPI tasks and MPI diagnostic tests, CCP_NODES is used to pass the list of allocated nodes from the HPC Job Scheduler Service to MS-MPI. For MPI tasks and tests that require over 450 nodes (the exact number varies according to the length of the node names assigned to the job), the list of nodes and cores can exceed the maximum number of characters (approximately 8,150 characters). When this occurs, CCP_NODES has a value of OVERFLOW, and the task or test fails to run with a corresponding message in the MS-MPI output. The workaround is to provide the list of node names to MS-MPI by using the /machineFile argument of the mpiexec command. For more information, see On large clusters, MPI jobs that include a large number of nodes can fail, including MPI Ping-Pong diagnostic tests.
CCP_TASKINSTANCEID	The numeric ID of a subtask. The HPC Job Scheduler service assigns this number when the task is created. The following task types have subtasks: Parametric Sweep, Service, Node Preparation, Node Release. Note All subtasks have the same TaskID.
CCP_RUNTIME	The amount of time that the task is allowed to run.
CCP_TASKSYSTEMID	The globally unique identifier for this task. This internal identifier is unique across all tasks in the cluster.
CCP_RERUNNABLE	If the task runs and fails due to cluster problems (such as a node becoming unreachable) and this variable is True, the HPC Job Scheduler service attempts to rerun the task. If it is False, the task fails after the first run attempt fails.
CCP_TASKCONTEXT	The context of the task that is used by MPI or other applications to communicate with the cluster.
CCP_WORKDIR	The working directory for input, output, and standard error files to be used while the task runs. The default is %USERPROFILE%.
CCP_MPI_WORKDIR	The working directory for input, output, and standard error files to be used while the task runs. The default is %USERPROFILE%.
CCP_RETRY_COUNT	The number of times that the job owner attempted to resubmit the task (including corrections to failed validation) and that the HPC Job Scheduler Service attempted to retry the task after the task was preempted or failed to run due to a cluster problem, such as a node becoming unreachable.
CCP_COREIDS	The core numbers of the local system physical cores assigned to the task. This is a space separated list.
CCP_NUMCPUS	The number of resources assigned to the task.
CCP_OWNER_SID	The Windows Security Identifier of the job owner.
CCP_ENVLIST	The environment variables that were set for this task by the person who created the task.
CCP_EXCLUSIVE	With a value of True, no other tasks can be run on a compute node at the same time as the task.
CCP_TASKID	The numeric ID of the task. The HPC Job Scheduler service assigns this number when the task is created. Note If a task contains subtasks, all subtasks have the same TaskID. Subtasks are further identified by the task environment variable CCP_TASKINSTANCEID.
CCP_JOBID	The numeric ID of the job to which the task belongs. The HPC Job Scheduler service assigns this number when the job is created.

The following task environment variables are set if the job owner specifies the corresponding job or task property in the job description:

^Top of page

Environment variables for diagnostic tests

Windows HPC Server 2008 R2 provides a set of built-in diagnostic tests that help HPC cluster administrators check that their cluster is working properly. The HPC Diagnostics Service creates up to three jobs for each test run, one for each stage (PreStep, RunStep, PostStep) for which the test explicitly defines a command. The HPC Diagnostics Service sets a series of job-level environment variables to help coordinate the test run.

To view the job-level environment variables that were set for a diagnostics job

1. In HPC Cluster Manager, in Diagnostics, click Test Results.

2. In the result list, select on or more test results, and then in the Actions pane, under Pivot to, click Jobs for the Tests. The view pivots to a filtered list of jobs in Job Management.

3. Double-click a job to open the View Job dialog box.

4. In the View Job dialog box, select the Environment Variables tab to view the variables that were set in the context of that job.

The diagnostics framework is extensible, enabling cluster administrators and partners to create custom diagnostic tests. You can use the diagnostics environment variables to help define your own custom tests. For example, a PreStep that adds tasks to a RunStep job can use the environment variables as follows:

for %%a in (%DIAG_NODELIST%) do (job add %DIAG_RUNSTEP_JOBID% /requirednodes:%%a hostname.exe)

For more information, see Diagnostics Extensibility in Windows HPC Server 2008 R2 Step-by-Step Guide.

The HPC Diagnostics Service can set the following variables as job-level environment variables for the clusrun jobs that the service creates for the diagnostic test run:

^Top of page

Additional references

• cluscfg listenvs

• cluscfg setenvs

• Get-HpcClusterProperty

• Set-HpcClusterProperty

• Environment Variables for the mpiexec Command