Tag Archives: Patterns

The Importance of Patterns in Messaging

Organization who dealt with scalability challenges successfully such as Google, eBay, Amazon and Yahoo, went through architecture changes and eventually reached a similar pattern: Asynchronous event-driven design.

Claim Check – Handling large file sizes

Messaging is most applicable to fast, frequent use cases of data transfer and notifications. Transport of large files is not, however the Claim Check Enterprise Integration Pattern (EIP) is very well suited for this.

How can we reduce the data volume of message sent across the system without sacrificing information content?

  • Store message data in a persistent store that has HA and is highly scalable
    • A NOSQL store is more appropriate than a relational database
      • Document Store: If you choose to store the file itself
        • CouchDB – REST protocol, JSON API
      • Key Value: If you choose to store the location URI of the file
        • Riak – REST protocol, JSON API
  • Pass a Claim Check (unique key) to subsequent components
  • These components use the Claim Check to retrieve the stored information

The Claim Check key allows you to replace message content with a claim check for later message retrieval. This pattern is very useful when message content is very large and would be too expensive to send around. They key here is that not all components require all information – some just need the key, while others need the actual data in the file.

It can also be useful in situations where you cannot trust the information with an outside party; in this case, you can use the Claim Check to hide the sensitive portions of data.


Patterns of Scalability and Pathways in Systems

Note: this is just a draft – I was trying to fall asleep, started to think, and this is what I thought about.

In college I wrote a 300-page senior thesis entitled, “Energy Pathways In Biological Systems”. It was within the context of genetics to microbiology up through complex pathways of micro-climates to ecosystems and even pathways of migratory animals (Arctic Wolves that cover at least 1,000 mile territories to Terns and Whales that have annual migration patterns covering half the earth). For each there is movement of elements in space and time. I had a blast researching and writing it but my fascination revolved around that shared concept over seemingly vast discrepancies of scale that were actually sharing massive similarities, being only sizable to other scales, can only be in a relative framework of complexity.

Think of systems as an atom. There are layers or levels and activity going on all the time. Now think of this atomic model with pathways repetitively used for resources to move, kind of like corridors. Resources behave differently from other resources, thus the corridors are different, the speed of motion is different and the size is different. Also production and consumption of those resources is different. Entropy works differently based on environment, among other factors.

The pathways of genetic information through a cell move in a seemingly small scale to Nitrogen pathways in a rain forest but the complexity in a cell looking down to the smaller elements in that system is great. Also in a rain forest, Nitrogen molecules and everything they interact with as they move through that system, looking up to larger components, are equally great and yet looking down to the components that amass that system we see the same thing.

Think about that – scale, if only quantifiable by the scale of other systems, is relative. So what could lend to differentiation? Complexity could be an important part of the equation. So what about this – System A is larger and has more components than System B. System B is less complex than System A. If both systems are replicated many times and distributed which may fare better? Hard to say with such a limited theoretical idea but what about Okham’s Razor – the simplest way is the best, essentially. In mechanics, the less moving parts, the less points of failure. A human is a complicated system, a virus is a very simple system and yet a virus can so easily attack the more complicated system. Cells replicate very quickly, and each new cell gets its own copy of the genetic instructions the original parent cell had. I’m rambling but just trying to give some simple examples to think about.

So how do we properly think about scale in systems? What can we learn from successful patterns of scalability that are all around us?