More Thoughts Around Apache Kafka – FIX Logs
Earlier this year I blogged about Raptor (Sungard) and real-time analytics. This posting follow on with a few specific Kafka thoughts:
- Kafka’s design page provides a few use cases, around “activity streams”. Of interesting is the list of companies mentioned that are using Kafka – in particular Metamarkets, a company I have blogged about before in the big data analysis space.
- The reference to log files is of particular interest:
Activity stream data is a normal part of any website for reporting on usage of the site. Activity data is things like page views, information about what content was shown, searches, etc. This kind of thing is usually handled by logging the activity out to some kind of file and then periodically aggregating these files for analysis
- The above lead to the thoughts around FIX messages, and specific the type of product that are commercially available from various vendors e.g. B2BITS with its FIX log analyzers
- OnixS FIX Analyser is again similar, monitoring log files
- All the above leads to leveraging Kafka to process FIX log files in real-time.
- The Log4j appender as show here should make things easy from an application perspective
- Likewise the Hadoop consumer offers some interesting possibilities
Further, assuming you now have this data, it maybe interesting to “count” FIX messages, which possibly leads us to data cube (backed by HBase).
Storm also offer interesting possibilities with FIX messages – another posting.