•January 8, 2016 • Leave a Comment
Scott Rosenberg “Can an Arcane Crypto Ledger Replace Uber, Spotify and AirBnB?” recent article is worth a read in my view. Specifically around the “d-word phrases”: Distributed Autonomous Corporation (DAC), Distributed Autonomous Organization (DAO), Distributed Cooperative Organization (DCO), etc. La’Zooz is most interesting, and something to watch, likewise Ujo. In many ways the article paint a possible future picture, which from a software engineering perspective is most interesting to say the least.
Tim Swanson is spot on with his remark about everyone wanting to put everything on blockchain – blockchain is the new silver bullet.
Finally, given the above future view, its probably also worth taking a look at Open Bazaar – the much speculated killer blockchain application.
•January 7, 2016 • Leave a Comment
Continuing down the ELK road, I ran into the issue of internal log files having internal network addresses, which when using the Kibana map facility, isn’t very helpful. However, thanks to a number of articles and posting on the web, this is completely solvable. Vlad Ionescu posting is particularly useful, leveraging the GeoLite City data. Another solution, in many ways a quick fix, is offered by Jim Cheetham. Finally, this posting offer a view on the geoip logstash properties to set to specify the GeoLite City database location.
•January 6, 2016 • Leave a Comment
A vast majority of software projects start out with no though as to what should go into log files. Log files are effectively a 3rd class citizen in the software development thought process. I’ve seen a few companies mandate certain data in log files for projects to ease the transition from development to production – handover to support (both infrastructure and application). Having spent some time using ELK, ingesting log files from applications that really aren’t structured very well, I now believe that ELK is an easy way for development teams to eat their own dog food.
What follows are a few points that I found useful:
- Get familiar with grok fast when using logstash
- When I started out with ELK, I unfortunately ingested log files that didn’t get grok’s appropriate. Deleting document is thus helpful as a query.
- Index’s are important :)
•January 4, 2016 • Leave a Comment
Christina Camilleri has an interesting presentation from QCon on “Securing Code Through Social Engineering”. Christina provides a few real world stories around social engineering (7mins) – password reset over the phone. Secure code has improved over the years due to publicity of issues from large companies like Microsoft and Adobe, and also thanks to the checklists, libraries, frameworks and books written about secure software engineering. However, social engineering can bypass all of these code improvements.
Be careful when your banks rings you :)
•December 15, 2015 • Leave a Comment
Great presentation from Erran Berger, Head of Content Engineering at LinkedIn. Discussion centres around stream processing, personal data and cache invalidation & replication conflicts. Take aways:
- Kafka – data is replicated (best and worst feature because it causes re-processing of messages)
- Kafka and DB replication turned out not to be the LinkedIn solution – due to duplication.
- The solution was to process user notifications in the data centre in which the user is attached too via a sticky routing service. Each data centre has a queue to route messages to other data centres.
- Process Kafka messages only once, else you’ll go down a painful road :)
- Replicate data only as required. Personal data doesn’t need to live everywhere in LinkedIn. Profile data does live everywhere.
- Router service decide which DB is accessed in which data centre using URN’s as the routing payload, and a catalog held in a DB for route the personal data request to the correct DB to retrieve the mailbox.
- Data centre connectivity needs to be low latency to aid this solution
- DB is shard (1000’s) for the 95 TeraBytes :)
- Cache invalidators
- In the case of LinkedIn, the cache invalidator listens to DB writes to invalidate the cache, leveraging the DB replication across data centres
- Soft deletes
- Custom conflict resolution if you can’t resolve in the data model/actors
•December 14, 2015 • Leave a Comment
Excellent presentation title :) “13 Ways Designers Screw Up Client Presentations”. If you can’t set context and present your ideas, your pretty much dead in the water. Watch the presentation, and read the book. Probably worth a watch/read by anyone that needs to present to clients – standard consulting stuff. High up on my list is the second bullet points in Mike’s article, “Not getting off your ass”:
Being on your feet will allow you to walk from person to person as they ask questions, simultaneously making you look more confident and allowing for more intimacy.
It should go without saying that you dressed nicely and your hands are out of your pockets
•December 12, 2015 • Leave a Comment
Stream Processing in Uber is work a watch over on InfoQ. Events emitted by applications, and processed by Apache Samza via consumption from Kafka. Storage of data in Elasticsearch. Which leads to a Lambda architecture, once Spark is added to the Uber architecture. Very informative talk.