Revealed: Windows Server HPC 2008 R2 F# Market Risk POC
So here in all its glory is my latest Proof Of Concept (POC). Leveraging everything I have blogged about previously, it’s time to see if we can move from the Excel RunnerHPC world😉 The diagram below essentially provide a high level architecture of what I hope will provide a useful guide to leveraging Windows Server 2008 HPC to calculate Market Risk.
Taking a feed from my simulated market (and the Market Strategy Engine POC) the trades will be submitted to the HPC cluster, where the F# cashflow/PV code I blogged about a few months ago will run as a service, crunching the trades, and generating PV numbers. Obviously to calculate the cashflows I need appropriate market data and curves. For this initial POC, I’ll pre-populate AppFabric caching with this data, and pass the velocity object keys to the HPC SOA with the trade data. A further POC will look at snapping this data, and updating Velocity at specific intervals, hence forcing portfolio recalculations – real world scenario.
The need for the HPC custom broker has been discussed in a previous posting, so I won’t re-iterate again. Likewise, the design of the Real-Time Cube (RTC). The VelocityClient on each HPC node is however important to discuss.
The HPC cluster needs to be party stateless, and partly stateful. This is driven by the need to reduce the amount of data sent around the LAN coupled with the type of Market Risk processing that will be undertaken. Stateless is probably more common on a cluster, since the process running the job/service will usually be ripped down on completion. Even if the process isn’t ripped down, one has to cater for the instance where the process crashes for some reason. Hence in stateless clusters, all data is shipped to each node, each time, but this is expensive.
The reason we need stateful nodes is when we move into the world of snapping market data/curves, and re-calculating portfolios. Given that we have snapped, we know that for x trades/positions/portfolios, certain data will be identical. Hence to reduce the expense of sending data, and improving performance, I’m planning to run a VelocityClient Windows Service on the Market Risk nodes. This Windows Service would do exactly what the service name implies, act as a node VelocityCache, drawing down certain a specific set (probably by currency) of market data/curves to nodes for a duration, and then flushing the service. The VelocityCache could be done away with if we kept the node service process (F#) running indefinitely, but as I said earlier this doesn’t cater for the process crashing, which could happen😦. Obviously not all nodes will cache the same market data/curves – as discussed in a previous posting.
The data coming out of the HPC cluster will be used to populate the RTC (previous POC), which also leverages Windows Server AppFabric caching, and hence provide cache notifications which via a streaming server (Nirvana in the case of this POC) update a Silverlight 4 RIA in real-time.
I’m sure there are flaw in this HPC design, but for the moment it appear viable, and worthy of a POC. Obviously I have a few other interesting idea up my sleeve post this POC, some of them are unfortunately dependant on Microsoft giving me access to some new technology😉