More Thoughts Around Apache Kafka – FIX Logs

Earlier this year I blogged about Raptor (Sungard) and real-time analytics.  This posting follow on with a few specific Kafka thoughts:

  • Kafka’s design page provides a few use cases, around “activity streams”.  Of interesting is the list of companies mentioned that are using Kafka – in particular Metamarkets, a company I have blogged about before in the big data analysis space.
  • The reference to log files is of particular interest:

Activity stream data is a normal part of any website for reporting on usage of the site. Activity data is things like page views, information about what content was shown, searches, etc. This kind of thing is usually handled by logging the activity out to some kind of file and then periodically aggregating these files for analysis

  • The above lead to the thoughts around FIX messages, and specific the type of product that are commercially available from various vendors e.g. B2BITS with its FIX log analyzers
  • OnixS FIX Analyser is again similar, monitoring log files
  • All the above leads to leveraging Kafka to process FIX log files in real-time.
  • The Log4j appender as show here should make things easy from an application perspective
  • Likewise the Hadoop consumer offers some interesting possibilities

Further, assuming you now have this data, it maybe interesting to “count” FIX messages, which possibly leads us to data cube (backed by HBase).
Storm also offer interesting possibilities with FIX messages – another posting.


~ by mdavey on August 8, 2012.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: