Issues with HPC 2008 R2 SOA Brokers And ISchedulerNotify
It appears that if you use the default Broker that comes with HPC 2008 R2, you get a durable broker, and can use DurableSession. However, move to a custom broker and you appear to be unable to leverage the default broker durability, hence your completely on your own 
Goodbye DurableSession
The HPC SDK has examples from a the Scheduler/Job/Task and HPC SOA. I was hoping that tasks would be more that what they provide, but essentially they are just a way of executing an application on a node. I’ll explain more soon as to what I was hoping to use tasks for – specially around pre tasks of a job.
In the HPC SOA world, you appear to get one broker per session. Each node in the cluster runs a Windows service – HPC Node Manager Service (HPCNodeManager.exe) which I assume receives the work payload instructions that need to be run on the node. In the case of SOA this would be the service to load, coupled with the payload to invoke on the service e.g.:
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://www.w3.org/2005/08/addressing">
<s:Header>
<a:Action s:mustUnderstand="1">http://tempuri.org/ISimpleService/Generate</a:Action>
<HPCServer2008_Broker_UserDataNS xmlns="http://www.microsoft.com/hpc">0</HPCServer2008_Broker_UserDataNS>
<HPCServer2008_Broker_ClientIdNS xmlns="http://www.microsoft.com/hpc" />
<a:To s:mustUnderstand="1">net.tcp://lab49hpc:9091/SampleBroker</a:To>
</s:Header>
<s:Body>
<Generate xmlns="http://tempuri.org/">
<max>10000</max>
</Generate>
</s:Body>
</s:Envelope>
The node Windows service appear to load HpcBrokerWorker.exe that would itself load the SOA service. I suspect the HpcBrokerWorker.exe is ripped down at the end of a session. Brokers are hence effectively a WCF service listening to events.
The CustomBroker sample in the HPC 2008 R2 SDK is worth looking at, specifically the usage of ISchedulerNotify. ISchedulerNotify appears to provide a callback mechanism so you can monitor the task running in SOA land:
public void TaskStateChanged(List<TaskInfo> taskInfoList)
{
lock (this.dispatcherList)
{
foreach (TaskInfo info in taskInfoList)
{
switch (info.State)
{
// If task is changed to Running state and client dic does not have this task
// Create service client for this task and add it into the client dic.
case TaskState.Running:
if (!this.dispatcherList.ContainsKey(info.Id))
{
this.dispatcherList.Add(info.Id, new Dispatcher(this.sessionId, info.Id, info.Capacity, info.MachineName));
}
break;
// If task is changed to Canceling, Finishing, Canceled, Finished state and client dic contains the service client
// Remove the service client from the client dic
case TaskState.Canceling:
case TaskState.Canceled:
case TaskState.Finishing:
case TaskState.Finished:
this.dispatcherList.Remove(info.Id);
break;
}
}
}
}
Must read: Suggestions to avoid certain timeouts and exceptions when using SOA
