Helena Edelson

Akka, Spark, Scala, Cassandra, Big Data, Data Science, Cloud Computing, Machine Learning, Distributed Architecture

  • cloud

  • RSS Subscribe

  • Topics

  • Archives



    • @helenaedelson on Twitter
    • Helena Edelson on LinkedIn
    • GitHub Octocat

      helena @ github

      • Status updating...


Archive for the 'Erlang' Category

Routing Topologies for Performance and Scalability with RabbitMQ

Posted by Helena Edelson on 1st August 2011

Designing a good routing topology for a highly-scalable system can be like mapping a graph. Many things need to be considered, for instance the problem, constraints of the environment, those of the messaging implementation, and performance strategies. What we often run up against is a lack of flexibility and expressivity in fitting routing to our needs. Here is where RabbitMQ stands out.

Basic Concepts

Anyone familiar with messaging in general knows the concept of routing messages from A to B. Routing can be simplistic or quite complex, and when designing a routing topology for a scalable, complex system it must be elegant. Kept clean and decoupled, components can throttle nicely with varying loads. This can be expressed as a simple map or complex graph. In its simplest form a routing topology can be expressed as nodes, for instance hierarchical nodes:

For those new to RabbitMQ or AMQP (note that Rabbit works with many protocols including STOMP, HTTP, HTTPS, XMPP, and SMTP), here are some basic component descriptions:

  • Exchange The entity within the server which receives messages from producer applications and optionally routes these to message queues within the server
  • Exchange type The algorithm and implementation of a particular model of exchange. In contrast to the “exchange instance”, which is the entity that receives and routes messages within the server
  • Message queue A named entity that holds messages and forwards them to consumer applications
  • Binding An entity that creates a relationship between a message queue and an exchange
  • Routing key A virtual address that an exchange may use to decide how to route a specific message

For point-to-point routing, the routing key is usually the name of a message queue. For topic pub-sub routing the routing key is usually hierarchical in nature:

api.agents.agent-{id}.operations.{operationName}

In more complex cases the routing key may be combined with routing on message header fields and/or its content. An exchange examines a message’s properties, header fields, body content, and possibly data from other sources, then decides how to route the message. A binding pattern derived from the above routing key idea might look like api.agents.*.operations.* where we bind exchange E1 to queue Q1 with binding pattern api.agents.*.operations.* so that any messages sent to E1 route to Q1 if their routing key matches the binding pattern.

A Rabbit broker is structured differently than a JMS Broker. Each RabbitMQ server is comprised of at least one node (broker), or more typically, nodes in a cluster. Each node has a default virtual host, “/”, and further virtual hosts can be created such as “/develoment”. Rabbit virtual hosts are like Tomcat virtual hosts and partition broker data into sub-sets. Within these virtual hosts are exchanges and queues. When a user connects with its credentials, it is connecting to a virtual host on a Rabbit node.

Here we connect to a Rabbit node, declare an exchange to publish to, a queue to consume from, a binding pattern, then publish a few messages, using the RabbitMQ java client api:

package org.demo.simple.rabbit;

import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.QueueingConsumer;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

public final class RocketSender {

 public void sendRockets() throws IOException {
     List<String> rocketsWithRoutings = new RocketRouter().build();

     Connection connection = new ConnectionFactory().newConnection();
     Channel channel = connection.createChannel();

     String rocketExchange = "rockets.launched";
     channel.exchangeDeclare(rocketExchange, "topic");
     String rocketQueue = channel.queueDeclare().getQueue();
     channel.queueBind(rocketQueue, rocketExchange, "galaxies.*.planets.*");

     for (String rocketTo : rocketsWithRoutings) {
         channel.basicPublish(rocketExchange, "galaxies.*.planets." + rocketTo, null, rocketTo.getBytes());
     }

     channel.close();
     connection.close();
 }
}

A simple consume of “landed” rockets could look like:

QueueingConsumer queueingConsumer = new QueueingConsumer(channel);
channel.basicConsume(rocketQueue, false, queueingConsumer);

int landed = 0;
while (landed < launched) {
    QueueingConsumer.Delivery delivery = queueingConsumer.nextDelivery();
    channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
    String rocketLanded = new String(delivery.getBody());

    if (rocketLanded.equalsIgnoreCase("Alderaan")) {
        System.out.println("That's no moon, that's a space station.");
    }
    landed++;
}

The Problem

In considering what routing strategies perform best in scalable environments where performance itself can also be improved, there many options. One of the great things about messaging in general is the variety of configurations available, and figuring out the right ones that solve both current and growing requirements.

To keep things simple let's consider two strategies:

  1. Highly-partitioned routing with hierarchical routing keys, fewer topic exchanges
  2. A larger number of direct exchanges and queues with far less routing partitions

Each scenario follows this use case: each application that must scale is both producer and consumer:

Producer Consumer

Where To Start

It is a good idea to take stock of your environment and its components before delving into a routing solution that will scale cleanly and efficiently over time. For example, what lends to scaling? Generally, decoupling, distribution, asynchrony, parallelism, levels of abstraction and indirection to name a few. Then consider what elements are current or potential bottlenecks. It is a basic principle that high traffic/volume pathways require more efficient throughput or you incur risk of bottlenecks in your distribution. One exercise is to rank these in terms of traffic or as a heat map. Next, can you classify your traffic – are there overarching patterns, topics or similar message types, and what are the relationships? Now start to consider consolidation, how and where might efficiency be improved, and apply tested patterns that resolve those heat points, decouple for scale, and increase performance.

General Routing Considerations

All exchange types behave differently. Here are a few general rules:

  • If you have a finite domain of routing keys in an application's graph then many fanout exchanges might be the right fit  (1:1 mapping of exchange per routing  key)
  • If you have a potentially infinite number of routing keys, consider topic exchanges
  • For topic routing, performance decreases as the number of bindings increase
  • Fanout exchanges are very fast because they have no routing to process yet if bound to a large number of queues that changes
  • Direct exchanges are a faster form of topic exchanges, provided you do not need the wild card
  • Troubleshooting problems across 100,000+ queues could be tedious versus a topology with more bindings, fewer exchanges and queues
  • A very high number of exchanges and queues take up more memory which may be significant but this really depends

As of RabbitMQ 2.4.0, released March 23, 2011, a new topic routing algorithm optimization is available that is 60 times faster at peak than the previous topic algorithm. Due to this, one recommendation is to go for less exchanges and queues, and more routing because the time increase is now minimal.

Performance

What is Cheap?

In terms of memory cost, exchanges and bindings. In Erlang, which RabbitMQ is built on, each node (broker) is a process, as is each queue. By default the Erlang VM process limit is set to 1M, which can be raised. However, an exchange is not a process for scalability reasons, it is simply a row in RabbitMQ's built-in Mnesia database. In a cluster, declaring an exchange causes it to appear on all nodes of the cluster, while declaring a queue creates it on only one of the nodes. This explains why exchanges survive node restarts or creating a node in a cluster, yet queues do not.

Be wary of binding churn. In strategy number 2, if you create many new queues and their bindings, whenever consumers attach you might run into problems. For instance, given exchanges E1...En to which many messages are being published, whenever consumer Cm connects, it creates bindings from its own queue to all of E1...En which may cause problems, depending on the rate of connections.

To alleviate binding churn, consider exchange-to-exchange bindings, new as of version 2.3.1. Each consumer could have its own secondary exchange Ym which must not be auto-delete. Then bind all of E1...En to Ym. In this way these bindings always exist. In this scenario, whenever consumer Cm connects it simply has to declare its queue and bind that queue to Ym. If Ym is a fanout exchange, it will be very fast and reduce the binding churn rate to 1 per connection, rather than potentially n per connection.

Exchange-to-Exchange Binding

Use Cases

Exchange-to-Exchange Scalable Use Case

Consider a server application with autonomous agents. Each agent is on a virtual machine that is part of an elastically-scaled system. As each agent starts up it sends a message to the server that it is online, followed by many other messages such authentication and data transfer. If we have 1,000 agents, each declaring 50 direct exchanges, queues and bindings, then each agent must know the server's queues in order to fulfill the binding contract on queue.declare operations. That is not a scalable solution.

Now consider creating shared topic exchanges: one exchange for the agent to server pathway,  another for the server to agent pathway, and a third to handle unauthenticated agents which routes only to those queues that do not require security. Now we partition with binding patterns, message routing keys and bring one set of those up for each server to be shared by all agents that connect to it. Then, in its simplest form, as each agent comes online it declares a private exchange and queue, and binds its exchange to the shared topic exchanges.

Our relationships are now expressed by exchange-to-exchange mappings which reduces churn rate and decouples agents from having to 'know' the server queues. Using this pattern the system is  clean, decoupled, and scalable.

Basic Routing Topology

Elastic-Scaling Use Case

Let's take the previous scenario a step further. We are already using topic pub-sub routing over scenario 2: many direct routings. Now let's say the system requirement bumps up to scale clusters of our server application in a data center with 50,000 or more agents. How can we throttle varying loads?

The authenticated client exchange routes messages from agent to server. It handles all operations publishing messages to singular-consumer queues, including those producing the highest frequency of messages. This is a potential bottleneck under the current topology with roughly 60,000 messages per minute for 10,000 clients or 86,400,000 messages per day. This is easily resolvable, RabbitMQ can handle over 1 billion messages per day depending on your configuration, for example whether or not you are persisting messages.

Our server applications are running a RabbitMQ cluster. Remember that in a cluster, declaring an exchange causes it to appear on all nodes, while declaring a queue creates it on only one of the nodes, so we have to configure a solution.

Load-Balancing Between Producers and Consumers

To efficiently handle these potentially very high loads as more client applications (Agents) come online, we can modify this topology in several ways. First, an optimization from the above configuration to load-balance messages across a Rabbit cluster. We can create one queue for each node in the Rabbit cluster. If we have four nodes, for each high-traffic queue we create hfq.{0,1,2,3} for that operation. Now each agent can randomly pick a node by a number between zero and three, or a more sophisticated round-robin implementation, to publish to. With RabbitMQ there are RPC calls or you can use the Rabbit management plugin to GET the number of nodes, which you can use in your round-robin algorithm.

Worker Queues With Round-Robin Dispatching

Worker Queues, or Task Queues, are generally used to efficiently distribute time-consuming tasks among multiple workers and easily parallelise work. In addition, this topology applies to eliminating the need to execute resource-intensive tasks and having to block until they complete. Running several worker queues allows these tasks to be distributed among them.

AMQP Work Queues, Round Robin Load Balancing

With Work Queues, by default, Rabbit uses a round-robin distribution method, sending each message to the next consumer in sequence. Each consumer receives roughly the same number of messages. If you declare a queue and spin up 3 competing consumers, bind them to the exchange, and send 20,000 messages, messages zero will route to the first consumer, message one to the second, message two to the third and so on. If we begin building up a backlog of tasks, we can simply add more workers, allowing the system to scale easily.

Performance

Memory

Neither option above will necessarily induce high load in RabbitMQ. There are no hard limits to the number of exchanges and queues, one can create, and running 100,000 queues on one broker is fine. With the right tuning and enough RAM you can run well over a million queues.

RabbitMQ dynamically pushes messages to disk to free up RAM, thus the memory footprint of a queue is not dependent on its contents. After a queue is idle for 10 seconds or more it will "hibernate" which causes GC on that queue. As a result, the amount of memory a queue requires can dramatically shrink. For example, it might be possible for 1000 empty, idle queues to take up 10MB of RAM. When they're all active (even if empty), they might of course, depending on memory fragmentation, consume much more memory. Forcing them back into hibernation to test behavior is difficult because the Erlang VM does not hand back memory to the OS immediately.

You can, however, observe a large process that hibernates and has very fragmented memory because the amount reclaimed can be sufficient to force the VM to hand back memory to the OS. If you run a test that steadily increases the memory footprint of Rabbit you could observe the effect of hibernation on idle processes as it reduces the rate of increase of memory use.

Erlang is a multi-threaded VM which takes advantage of multiple cores. It presents green threads to the developer which are called 'processes' because unlike threads, they conceptually do not share an address space.

Transactions

Transactions on 10,000 messages can take as along as four minutes to publish. A new RabbitMQ feature called Publisher Confirms is more than 100 times faster than the same, but transactional, code. If you are not explicitly required to implement transactions but do need the verification you might consider this option.

The Take-Away

Here are some final takeaways to help you squeeze the greatest performance gains out of your implementation:

  • The new topic routing algorithm optimization is 60 times faster at peak
  • Topic binding patterns using wildcards '*', which matches a single word, is much faster than '#', which matches zero or more words. Wildcard '#' takes longer to process in the routing table than '*'
  • Exchange-to-exchange bindings improve decoupling, increase topology flexibility, reduce binding churn, and help increase performance
  • Exchanges and bindings are very light weight
  • RabbitMQ Publisher Confirms are more than 100 times faster than AMQP transactions
  • After a queue is idle for >=10 seconds it will "hibernate", inducing GC on the queue, resulting in a dramatic decrease in memory required for that queue
  • Worker Queues help parallelize and distribute workloads
  • Distributing worker queues in the Rabbit cluster helps scale
  • Load-balance your topology

This is by no means a thesis on the subject, there are indeed many more patterns, topologies and performance details to consider. A strategy, as always, depends on so many factors but I hope this encapsulates enough to help or at least get one thinking in the right directions.

Get It

RabbitMQ Source on GitHub
RabbitMQ Binary Downloads and Plugins
Erlang Downloads
Spring AMQP API for RabbitMQ in Java and .NET
Hyperic to monitor RabbitMQ
Maven


<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
<version>${rabbitmq.version}</version>
<exclusions>
<exclusion>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
</dependency>

Share/Save

Posted in AMQP, Broker Topology, Cloud, Erlang, Messaging, Open Source, Performance, RabbitMQ, Scalability | No Comments »

Routing, Performance and Scalability with RabbitMQ

Posted by Helena Edelson on 7th April 2011

My latest post on the Spring Team blog
Routing Topologies for Performance and Scalability with RabbitMQ

Share/Save

Posted in AMQP, Cloud, Erlang, JMS, Messaging, Open Source, Performance, RabbitMQ, Scalability | No Comments »

Why RabbitMQ for Cloud?

Posted by Helena Edelson on 2nd February 2011

We know that Messaging itself affords moving from a SQL -> Database -> File System model and into the Network layer itself with:

  • Logical decoupling
  • Physical decoupling
  • Temporal decoupling

but I’ve been asked why I would choose implementing RabbitMQ in a cloud solution. This is a big topic, and there is a great deal more I’d say conversationally but for a relatively quick overview let’s start from the ground up.

Why AMQP?

AMQP is an open, wire-level binary protocol for Message Orientated Middleware that was created by and continues to developed collaboratively for users in mission-critical real-world scenarios. AMQP solutions are deployed to high performance clustering and grid computing problems and is developed for typical business scenarios:

1. distributes data to hundreds of clients in real time across a global network
2. accepts workloads to process from those clients and handles them reliably in a clustered transaction processing engine
4. connects to other partners over the Internet to exchange data
5. carries out all these functions in a secure manner which would pass an audit
6. monitors application traffic to build up a real-time activity

AMQP implementations offer fire and forget where you can absolutely trust that your messages will reach their destination, extremely fast delivery of data to a large number of consumers (publish sub-event notification), and
secure, firewall friendly file transfer.

“AMQP is a good fit for cloud MaaS and other wide-area, Internet-based messaging purposes, because it defines the wire protocol and communication semantics without restricting the details of the messaging system implementation or the API. AMQP’s goal is to enable interoperability among disparate vendors’ platforms.”
- Roy Shulte (Gartner), Review of AMQP and other protocol options.

More: amqp.org

Why Erlang?

All major implementations of AMQP such as Open AMQP, Qpid and RedHat MRG are built on a foundational technology. All but one are built on C++. RabbitMQ is built on Erlang. Erlang insures concise binary matching, has built-in support for concurrency, distribution and fault tolerance, has a built-in distributed database, is open source, and operates on many platforms. Erlang is also a runtime environment, similar to the java virtual machine, allowing code compiled on one architecture to run anywhere. The runtime system also allows code in a running system to be updated without interrupting the program.

More: erlang.org

Why RabbitMQ?

Many large-scale projects are using RabbitMQ for complex, scalable, cloud topologies, such as Observatories Initiative (OOI), NASA Nebula, Hiroku, Unique Identification Authority of India (200 million messages per day peak), and CERN because

  • It’s wicked fast and reliable
  • Supports security and transactions, persistence, client acks, and flow control
  • Widely proven and used heavily on Amazon EC2
  • It is open source with support services and several very active user lists
  • Hugely interoperable
    • RabbitMQ brokers run on many platforms and offer several clients (Java, .net and others), all which speak AMQP protocol to each other. Anyone can write a RabbitMQ client in any language: Ruby, Java, Spring, Python, C, C#, Perl, Erlang, Lisp, Haskell, PHP, _your_language_here.
  • Easily clusterable
  • Works with several protocols: AMQP, STOMP, XMPP, 0MQ
  • Highly-flexible topologies
  • Can work with Pacemaker and associated tools to provide High Availability
  • Supports several extension points via plug-ins, e.g. exchange types and backing queue storage
  • Provides management and monitoring of brokers
    • vFabric Hyperic
    • RabbitMQ command line tool and plugin for real-time activity and management
  • Messaging services hosted in the cloud with RabbitMQ as part of overall cloud infrastructure services are highly-supported

An open source RabbitMQ solution stack might look like this: Spring, Tomcat, Hibernate, MySQL, Hadoop, MapReduce, Chef, Apache httpd, RabbitMQ.

More: rabbitmq.com

Resource List And Use Cases

Share/Save

Posted in AMQP, Cloud, Cloud Ops, Concurrency, Erlang, Messaging, Open Source, RabbitMQ | No Comments »

JInterface Can not connect to Peer Node

Posted by Helena Edelson on 30th September 2010

I ran into a nasty issue with jinterface when making a rapid succession of calls at the start of the application. I need to move this to a caching solution of the connections today but here is a quick solution for anyone else hitting their head against the wall with this. Jinterface swallows exception causes and it is very hard to hunt down the true underlying error. You will see the same error message for n-number of things that can go wrong farther down the stack.

First I’ve not added a boolean setter for if you want to use unique each time or not, so that’s something you can add. Also there’s a connection monitor for shared connections. I’ve left out all error handling for brevity. And finally, also for brevity, just used one constructor using the cookie to keep this shorter.

public class ConnectionFactory {

    private String cookie;
    private String selfNodeName;
    private String peerNodeName;
    private OtpSelf otpSelf;
    private OtpPeer otpPeer;
    private OtpConnection otpConnection;
    private final Object connectionMonitor = new Object();

    public ConnectionFactory(String selfNodeName, String cookie, String peerNodeName) {
		this.selfNodeName = selfNodeName;
		this.cookie = cookie;
		this.peerNodeName = peerNodeName;
                initialize();
    }

    public void initialize() {
		// assertions...
		String selfName = this.selfNodeName + "-" + UUID.randomUUID().toString();

		try {
			this.otpSelf = new OtpSelf(selfName.trim(), this.cookie);
		} catch (IOException e) {
			//
		}
		this.otpPeer = new OtpPeer(this.peerNodeName.trim());
	}

	public OtpConnection createConnection() {
		synchronized (this.connectionMonitor) {
			if (this.otpConnection == null) {
				try {
					doCreateConnection();
				} catch (IOException e) {
					//
				} catch (OtpAuthException e) {
                                 //
                         }
                  }
		return this.otpConnection;
		}
	}

	public void doCreateConnection() throws IOException, OtpAuthException {
		synchronized (this.connectionMonitor) {
			if (this.otpConnection != null) {
                             otpConnection.close();
			}

                       this.otpConnection = otpSelf.connect(otpPeer);
		}
	}

	public void destroy() {
		resetConnection();
	}

	public void resetConnection() {
		synchronized (this.connectionMonitor) {
			if (this.otpConnection != null) {
				otpConnection.close();
			}
			this.otpConnection = null;
		}
	}
}

Share/Save

Posted in AMQP, Erlang | No Comments »

Installing Erlang and RabbitMQ on RedHat

Posted by Helena Edelson on 13th September 2010

I am setting up a thin-provisioned vm running RedHat to host erlang and RabbitMQ for our QA team to test my code. Having not done anything on RedHat (just Ubuntu) for a long time I had forgotton everything and yet it was still incredibly simple:

Find the latest release – this was at the time of my install. The path will give you the right idea for the current version:

yum install erlang

For some reason yum has a pretty old version so you may want to manually install it. Choose your RabbitMQ server version -  2.1.0 was the current for this install.

wget http://www.rabbitmq.com/releases/rabbitmq-server/v2.1.0/rabbitmq-server-2.1.0-1.noarch.rpm

rpm –install rabbitmq-server-2.0.0-1.noarch.rpm

Then I verified in a few directories that RabbitMQ was installed, and ran

/sbin/service rabbitmq-server start

Verify that your RabbitMQ node is running:

rabbitmqctl status

If you have no nodes up rabbitmqctl will run but tell you no nodes are up to report status on. Simply restart at least one node.

If you want RabbitMQ to start after everything else on startup/reboot do:

ln -s /etc/rc3.d/S99rabbitmq-server ../init.d/rabbitmq-server

Share/Save

Posted in Erlang, Messaging, RabbitMQ | No Comments »

RabbitMQ 2.0.0 Released

Posted by Helena Edelson on 25th August 2010

More details here.

Release posts are about all I have time for these days…been heads down for a month getting an amqp/rabbitmq thing ready for release.

Share/Save

Posted in AMQP, Erlang, Messaging, RabbitMQ | No Comments »

RabbitMQ and Erlang Intallation for Mac OSX

Posted by Helena Edelson on 11th June 2010

I’ve been a linux developer for years so I was all thumbs having to look up everything when setting up a new dev environment on a mac. Here’s some tips for setting up a local RabbitMQ server to play with (i.e. not using macports locally, not the EC2 install etc):

Requirements

  1. Apple XCode to compile Erlang with gcc. The XCode download could not take longer. Start it, go build an enterprise app in ROO, deploy it to the cloud. When you come back it may have completed.
  2. wget from http://ftp.gnu.org/pub/gnu/wget
  1. tar -xzf wget-{version}.tar.gz
  2. cd wget{version}
  3. ./configure
  4. make
  5. sudo make install

Install and build Erlang

First, Check for the latest Erlang GA Release and note the version.

  1. cd to the dir you wish to install into
  2. wget http://erlang.org/download/otp_src_{version}.tar.gz
  3. tar xzvf otp_src_{version}.tar.gz
  4. cd otp_src_{version}
  5. NOTE: run ./configure –help to see what opts you want to run with
  6. ./configure
  7. make
  8. sudo make install

Install RabbitMQ Server

First check the latest version of RabbitMQ Server and note the filename.

  1. wget http://www.rabbitmq.com/releases/rabbitmq-server/v2.3.1/rabbitmq-server-generic-unix-{version}.tar.gz
  2. tar xzvf rabbitmq-server-generic-unix-{version}.tar.gz
  3. Complete the install and config from http://www.rabbitmq.com/install.html#generic-unix, making sure the logs, clustering, data and config dirs are set up
  4. Start RabbitMQ: run rabbit../sbin/rabbitmq-server

Infinitely easier on Ubuntu.

Share/Save

Posted in Cloud, Erlang, Messaging, RabbitMQ, Spring ROO | No Comments »

Future Posts: Esper, RabbitMQ for Cloud Messaging

Posted by Helena Edelson on 9th May 2010

If I had more time I would add a few new posts on esper and rabbitmq for messaging in the cloud but I just got home from 2 weeks of coast-to-coast consulting and technology presentations. Pretty exhausting. I hope to have time to post on these and a few other fun technologies. I’ve got some code samples and tests I might post with that, plus some cools stuff on esper queries with Spring Integration.

Share/Save

Posted in Concurrency, Erlang, Java, Messaging, RabbitMQ | 2 Comments »

 
a2a_init("page"); a2a_init("page"); a2a_init("page"); a2a_init("page"); a2a_init("page"); a2a_init("page"); a2a_init("page");