Category Archives: Hadoop

Comparing and Choosing Technologies

Unable to sleep at 2:30 AM, I just read an interesting case study with a large company on how they have started to address Ad Tech applied to TV households vs mobile and web, with frequency capping to viewers, etc. Not surprisingly they use Kafka. It made me think of a situation I was in where someone brought up Solace versus Kafka.

When I listen to people that talk about any technology, I try to really listen to

  • Whether they are speaking from a place of personal experience or from what they have read and surmised from blogs by users that may or may not know what they are doing with that technology, may or may not have applied it properly, and so forth
  • If they have merely played around with that technology or do they have actual production experience
  • And if they have prod experience with that technology was it actually at scale, under real world load

Or, are they talking about a solution where required features are not yet completed – i.e. currently vaporware in a sense. These sort of situations must be taken with care, as opposed to proven ones.

For example, trying to compare Solace to Kafka is comparing apples to oranges. I have seen this sort of comparison attempted many times over the last decade that I have been in messaging, and it (a pun) is fruitless :)

They both serve a different purpose in an architecture, and while they overlap in some functionalities, Solace seems to fit more particular use cases – and handle those use cases very well. While some features are both out of the box with Kafka, and proven in production – at extremely high load and at scale.  No technology is a one size fits all solution. For example I know a production system that uses both an Aerospike cluster to solve some problems and a Cassandra cluster to solve others – both NoSQL stores serving different applications in different areas of the architecture. Taking this a step further, a deployment architecture for one technology, such as Cassandra, would have multiple clusters, one cluster for analytics, one cluster for timeseries, and so forth.

The concept I want to highlight is event-driven architecture, nothing new, and I’ve been working in this for years, but it is quite different from more conventional ETL data flows. Leveraging this architecture can both decouple cleanly and improve speed if done right. In ad tech if you do not respond in time, or correctly, you lose the bid and thus lose money. But this is only one part of the system. It’s not the high-frequency trading use case but it shares some similar requirements.

If you are promised features or an MVP by a team you may be a customer for, including being an internal customer, consider whether that team is motivated enough to have your team/company as a customer, and willing as well as able to do the work quickly to support it. For example, I would dig further into this with their product manager, project manager and engineering lead to find out if they have the a) roadmap, b) priorities, c) resources d) QA, test automation and perf engineers to do this. I wouldn’t want to sign up for it only to find they can’t deliver, or can’t deliver in the necessary time frame.

Things like hadoop or terradata are very nice but you can’t scale with them, and they are not applicable for streaming, event-driven architectures and workloads. You can however scale with Kafka and leverage Kafka to handle the ingestion stream to run analytics and respond back in-line, for example a system run on Kafka that can do distributed RPC, leveraging Akka (or Storm but akka is easier). You can do a lot in 20 ms. For example you can write a system where the consumer sees the produced data in 2 ms tops. A response via Akka remoting for example is another ms hop back. 1 ms response time. There are many ways to solve this of course, and here again, you can use Kafka – Storm, you can use Kafka – Akka and feed to Spark if you can do micro-batching and do not need very low latency analytics response. It all depends on your requirements – what are you solving for, your competition and your constraints.